Improve Your HighTech Deal Flow Management with airSlate SignNow

How to create outlook signature

(upbeat music) - [Lois] Hello, and welcome to the MIT System Design and Management Program's Systems Thinking Webinar Series. My name is Lois Slavin (clears throat) excuse me, and I will be your host for today. SDM stands, as I've said, for System Design and Management. It's MIT's master's program for midcareer professionals. It's offered jointly by the MIT Solan School Management and the School of Engineering, and it offers an education that focuses on integrating the technical, business, and sociopolitical components of complex challenges using systems thinking. Our graduates earn a master of science in engineering and management. The MIT SDM Systems Thinking Webinar Series features research conducted by SDM faculty, alumni students and industry partners. This series is designed to disseminate information on how to apply systems thinking to address, once again, the engineering, management, and sociopolitical components of complex challenges in virtually any domain. Recordings from prior SDM webinars can be accessed and viewed on demand at sdm.mit.edu. Today's webinar is being presented by SDM fellow Bryan Pirtle. He's a cofounder and chief technology officer of Builder AI, and formally worked for Gallo Winery. He's a real testament how, what he's learned at SDM can be applied in different industries. - [Bryan] All right, thank you Lois. Hello everybody and thank you for joining us today. So the agenda for today, first a little bit about me, and I'll go into sales as a system and what aspects of the system can an AI improve, and then building an AI product to improve sales, choosing an AI technology stack, and then finally, a Q&A. So first, a little bit about me. From a very young age, I was interested in computers and technology, and my passion for creating and hacking in gaming environments, and I mean, of course, white hat hacking, translated as I grew up into a career in software and engineering. I got into software and engineering professionally during my later undergraduate years, where I worked as a consultant building e-commerce websites and the firmware for embedded systems that could interface between the onboard computers in all major cars, and the mobile devices of the day, which were Palm pilots, laptops, et cetera, because smartphones actually didn't exist yet. Then I spent the next few years working for the largest winery in the world on real-time industrial control systems, big data acquisition, and IT infrastructure design. Now, 2013 was a turning point for me, 'cause I was fortunate enough to be admitted into the SDM program which allowed me to broaden my base of knowledge about business, engineering, and technology, where previously I'd really just been doing engineering work, and it positioned me to become an entrepreneur, which I'd always wanted to do. So I spent the next several years trying a lot of things out in a lot of different spaces, and failed a bunch of times, learned valuable lessons, before my now business partner came to me with the idea of tackling the sales space, for which he had some interesting insights. Now, after diving deeply into the sales cycle for about a year, and thinking about how technology could improve it, running some tests, we came up with Nova.ai, and today I'm gonna walk you through that process that we came upon for building an AI product for sales. So Building an AI Product to Improve B2B Sales. That's the title of this talk. Before moving forward, I'd like to go over a couple of important definitions. First, B2B sales, which is business to business sales, or businesses that are selling to other businesses. This is in contrast with a very different type, which is B2C or business to consumer, which, we will not be focusing on today. Now, in B2B sales, the hot approach right now is to do account-based marketing and selling. This means that if sales team's goal is to win an account out of their customer by finding and selling to people relevant context at that account. So second, what is an AI product? There are many definitions, but the most applicable to what we're gonna talk about is the following. A product that performs cognitive tasks that could previously only be done by humans. And the last part of this is building. How do we go about building an AI product for B2B sales? First off, a bit about building a product more generally. This actually came from "The Four Steps to the Epiphany," a good book on this topic by Stephen Blank. Now, we took the customer development process as a guide to building our product. You start on the left here with your customer discovery phase. We are trying to find a gap somewhere in some market that people want sold. Then you have to go ahead and build the product in the hopes of filling it. Now, this turns out to be really hard, so you are likely going to ping-pong between these two for a very long time, and get the feedback before you have an MVP or what we generally call a minimum viable product. After this, you can go into customer creation where you build a scalable sales process, and finally go into wholesale company building, and maybe eventually to IPO and beyond. So let's go back to the customer discovery phase of the process, and talk about how we at Nova.ai found gaps in the sales technology ecosystem and why we decided that AI was the right way to fill them. The customer discovery phase of building a product begins with figuring out what the current ecosystem looks like. You need to understand the status quo before you can improve it. The current ecosystem for sales technology shown here can be visualized as a cyclical progression. At the top left area, you'll see lead generation. Starts here where sales and marketing teams are trying to find appropriate contacts to reach out to. These contacts are loaded into a CRM, which is a customer relationship management system. This is the system of record or basically a database on steroids for sales teams. Basically, all data in the sales process is gonna filter through this system. Then there is an automation layer between the CRM and the actual customers and leads teams are trying to create relationships with. This automation layer is where most of the current startup innovation is going on, and it's also where AI can be most readily applied. Some typical tasks that this layer helps with are automation of scheduling, call logging, emailing, and CRM management itself. Now, worth noting here is that this entire flow is cyclical, because if you look at the top right here, leads become customers and existing customers can be upsold further to become new leads and ultimately new customers of more of a company's product offerings. So again, the backbone of this entire process, if you look at the bottom, is the customer relationship management or CRM system, so it's worth digging in a bit more here to understand what it does. At its core, a CRM is the system of record for the entire sales process, and it's effectively an extensible collection of objects with rich interactions among them. Most sales teams use the following objects. The contact object, which represents all the information that you as a salesperson know or sales team know about a particular person, who you might be able to sell to at some point, or who already is a customer of yours. Then you have a lead, which represents a person who has expressed some level of interest in the product or service you're selling, and it can vary in terms of how hot they are to buy. Then you have an opportunity, which is usually directly associated with a lead and represents potential revenue the sales was able to close the deal. Then you have an account, which is an account you're trying to sell to or already is your customer, and this is associated to all of the contacts that are inside of the account. And then finally, you have the activity, which is associated to all the other object types, and basically represents anything that a salesperson or let's say the contact or lead that it represents has done. For example, this could be sending an email, replying to an email, making a call, having a meeting, things like this. Now, there are a bunch of other and custom objects. It's a very rich system, but these are the main ones that most sales teams use, so we'll focus on those. So the foundation of this is a lot of information about people. So here is an example of what the data about a person looks like. Note all of the fields with relevant data about the lead that a salesperson might want to know, and especially the lead status fields, which represents the current state of the lead in a sales process. So the CRM helps manage this state machine as a lead progresses through the sales process. So now, we've talked a little bit about the sales system itself, some more on an archetypical B2B sales process. What does it look like? Now, we'll be focusing on the middle here, which is this blue part, 'cause that's the sales part. But the top and the bottom are also important in understanding the interfaces of the middle part, so those are included here as well. Now, this process can be visualized as a funnel, which most of sales and marketing can, and it starts the talk where marketing is operating. This phase is generally focused on fostering brand awareness and also generating new suspects, which are relevant contacts for sales. Now, sales development comes next, and they'll take the suspects from marketing and they'll also do some of their own lead generation to create and begin to work on leads to get them to a point where they can pass the hot leads to account executives, as what they like to call, SQLs or sales qualified leads. So usually a hot lead or SQL is ready to buy or very close to it, and becomes an opportunity for the AE to close, the account executive, they're the closers. So then after an opportunity closes and becomes a customer, the account is now a customer and all associated contacts are considered customers. Customer Success takes over from here. Now, in this process, we have a lot of instances of someone on our sales team, directly interacting with the contact, as it become a lead and hopefully a customer. Each of those interactions is gonna be captured by the CRM. So here, in the CRM, we can see a record of all of the activities from that funnel. This is the same lead object from earlier, just a different view, which has the activity feed, and we can see everything they did, every email they sent, every time they opened an email, every link they clicked, every time they were interacting in some way with our salespeople or content. Now, the other thing to note is that this shows not only when our salespeople did something, but also when the lead did something, and the specifics about the interactions between the two, which is very important to the sales process. Since we're gonna focus here on sales development in the middle of the funnel, what activities are these sales development reps actually doing? And what types of activities are they? Now, okay, the activities themselves, they mainly focus on these five. They focus on lead generation, they focus on CRM management, sending emails, picking up the phone and calling people, and qualifying process. Now, if you notice on the right here, there is a typical flow that a sales development rep will take from taking a contact or lead at the beginning and then going all the way to qualifying them and then passing them to the account executive. Now, there is a badge on the left of each one of these steps, and this shows whether it's a manual process, a semi automated process, or an automated process. After every manual or semi auto tasks that an SDR does, there is a required CRM update to keep it synchronized and modern automation platforms do a good job of doing this without manual intervention. Okay, let's go through this real quick, because this will set the stage for where an AI might apply. The first step, lead generation, is currently a fully manual process that SDRs have to figure out who is a good fit to contact, and then see them into the CRM. Then with data they do the first email, often called the cold email, which is actually a semi automatic process. Now, this is because the automation platforms of today allow users to create message templates with magic tags in them that get directly substituted with variables from the CRM data, so what does this mean? These things might be first name, company name, industry, things you saw from before. And you may yourself have received a message that starts with something like hello first name in curly brackets, which is what happens when the automation system messes up and embarrasses the sales rep. This is not what you want it to do, but it is semi automated because the sales reps are not actually typing most of the email content. Comes from the template, and the CRM variable substitution, and a complete but formulaic message can be created and sent, so the user doesn't have to write each one from scratch. And templates are often shared and reused within different sales teams. So the next few steps are all manual. Once a prospect responds and indicates interest, there is a scheduling phase and qualifying call with subsequent CRM updates. Then next, there is a back and forth of manual emailing while the sales rep prospects set up a meeting. While automated dialers do exist to carry out these meetings, digital transcripts of the qualifying call and any notes are not captured, so CRM update is only semi automated at best, and it still of course requires manual effort on the part of the sales rep to complete it. So after the qualifying call, many more manual emails will go back and forth as the sales rep finishes the qualification process and tees up this prospect as an opportunity for the account executive. And these are all captured inside the CRM as well. How and when a prospect becomes a sales qualified lead, an opportunity, is also currently a manual operation that's only done by human sales reps. What is an AI product? So, AI products perform cognitive tasks that could previously only be done by humans. So let's apply this to the sales development cognitive tasks, or tasks, that we just discussed. Well, turns out, that actually all of the manual and semi automated tasks can have AI applied to become fully automated. Now, one caveat to this. Humans certainly cannot and should not be completely removed from the sales process. So tasks that require us such as a phone call or closing process might just utilize the AI to provide new information and workflow relevant to the task completion. Now, you might ask why AI for sales development? We found all these manual tasks that could use AI, but is the juice really worth the squeeze? So, first of all, there's plenty of automation out there, but there is a general lack of intelligence platforms for salespeople. Most of the systems are automated, but they are pretty dumb, and everyone admits to this, so I'm gonna make some strong statements here about the future and things we learned and so why we think AI is worth applying to the sales process. Now, we spent a year studying it and running experiments and came to these conclusions. So first of all, highly personalized digital interactions are the future, and secondly, people want to buy from people they like and are like them. And as we explore this space and after several iterations on this customer discovery cycle that I spoke about earlier, we decided to focus on emails out of everything that we could possibly improve upon for sales development. This is for a few reasons. First of all, email's still the best channel for inter-company communication, as opposed to intra-company communication, where real-time systems like Slack and text messaging are great. They're great for internal communication, but they are still too intimate for today's sales communication, coming from outside the organization. And not to knock marketing, they send a lot of high-volume emails, and this is important, but a continual complaint we hear is that the email channel is being inundated by spam, or largely untargeted template-based and interpersonal emails. So AI can fill in the gaps here, and help ensure that every email that goes out is high quality. It's targeted and filled with relevant, appropriately personal and engaging content, a.k.a, and then typing email that you'd actually want to read. Additionally, it was clear that the average SDR was spending several hours per day, some would say up to half their day writing effective emails, and this was just not scaling with their expected deal flow for the organization, and perhaps most importantly, we found this small number of these high-quality personal emails work incredibly well in engaging prospects and moving deals forward. Now, why would you want to send out thousands of mass emails when you could send out hundreds or even less and get not just the same number of sales qualified leads but more of them? And ones that were even more highly engaged than before? Now, how good was it? Well, we ran a ton of experiments and it was incredibly effective to do it this way. So we ran experiments where we would hand-personalize emails in a variety of contexts and try a different, a lot of different tones of voice, and it was pretty complicated, but it was a controlled experiment, and then we checked the feedback, the opens and the click rate, and so the numbers on the right, the bulk email came from Mailchimp, and so out of their tens of millions of emails that they have sold, they get an average 23% open rate and a 4.2% click rate, and our hand-personalized ones, so we were writing them by hand but the goal, eventually, of course, was to have a computer write them, was this 3.5x more effective across-the-board. Now, there were some caveats to this. First, slightly warm leads worked far better than ice cold ones. So this made sense to us. Reaching out in a very personal manner to someone you already have some relationship with is likely to be met well, but if we don't know the person, it's much easier to sound disingenuous. Also, depending upon who the person was, what you keyed on and how you said it definitely mattered a great deal. So the resulting effect was very interesting, because it was found to be more polarizing than always positive. If it was applied incorrectly, highly personalized content could actually backfire on you and decrease your effectiveness. The takeaway, for us, however, was incredibly valuable. If the AI could be taught to understand the pitfalls and avoid them properly, it could achieve incredible results that normal salespeople on their own never could, and it could do in record time. So the goal was simple, but it turns out to be extremely difficult to achieve. So it was, the goal was to send the right email from the right sender to the right recipient at the right time. And after focusing on the personalization piece for a long time, which is about sending the right email, we learned that its effect can be multiplied again even further if the overall message is constructed in specific ways. And that this changed depending upon who you were sending to. Since most sales teams have a lot of reps on them, it is also very important to determine who on the team should be reaching out to a particular recipient at any given time. The AI is particularly well-suited to understand who is on your team, and then the unique ways particular individuals may be well-suited to connect with particular recipients. Now, we've been able to build an AI that is attempting to address the first two, but it turns out we couldn't even get to the second two. And that's because those are very difficult problems in and of themselves, to the right recipient at the right time, and there are companies dedicated to each one of those, so we decided that we stick with the first two for now. Okay, so to accomplish this goal, the AI has to be able to do some specific cognitive tasks, and it has to be able to recognize content from unstructured data. The AI can determine when it should send certain content and glean meaning classification, it's a classification problem, by analyzing prior unstructured email texts in the form of past sent email from senders, and then replies from email recipients, as well as full threads of email conversation. There's a rich set of data in there if you can actually recognize the context. Second is cleaning messy data and converting data into usable information. The AI needs to be able to take data from the CRM, and then pair it with new data it is able to find on its own, and structure this into a form that it can understand and apply, and finally, applying structured data in new contexts, and using feedback to increase goal attainment. The AI can take past messages and use the feedback, did the recipient open, engage, buy or not, to predict future success with similar email content and recipients. All of these cognitive tasks feed into a learning model designed to achieve the main goal of properly personalized email content generation, also known as writing a personalized email, so that AI writes an email. This is an example of what it actually looks like. The portions in blue were actually written by the AI. The datasheet on the right pane was assembled by the AI, not actually part of the message, but available to the salesperson, while the portions of the message in black were from a template written by a human. Notice that the AI is aware of a few things here. So it knows that Sally and Shirley share an alma mater. Great to see another Stanford grad in the industry, do you make it back to the farm often? I have my five year coming up. There is a relevant case study for Shirley's company, Loud Sound, and also that Sally has several customers in Shirley's space, Nano Soft and IQ and Google. The AI will also never just send an email like this out unless Sally wants it to. The typical process is that the AI will notify Sally that it's finished writing, and that it's her turn to review it and make final edits and tweaks to make it her own before giving the go-ahead to send. Over time, the goal is that the AI will become so smart that Sally trusts it, and she will let it send out many emails on her behalf in a fully automated way without review, because she knows it'll already be what she would want without requiring edits from her. So now that you've seen the output, how does the AI go about building such a message? First, it embarks upon an extensive data gathering phase where it uses dozens of public data sources, such as APIs, websites, articles, blogs, and public record database to collect a complete raw data set on the recipient. Then it uses this new data plus data from the CRM entered by the sender and cleans it into a cohesive and highly structured profile about the recipient. Some notable components of this profile are interests, achievements, and organization affiliations. Also worth noting is that the sender who's already a user of the system has had this process done previously, so the AI now has a profile for both the sender and the recipient. Now that it has the profiles, the AI determines if the sender and recipient have anything in common, we call this affinity recognition. Some examples below, alma mater, similar company, similar industry, shared interests, lives in the same place, as well as any notable recipient accomplishments worth mentioning. Some of these are patents, publications, recent promotion, honors degrees, et cetera. Finally, the AI uses all this information in its language structure model to construct language that, coupled with the template written by the human sender that you saw before, the black areas, creates a highly personal email message from the sender to the recipient. So every AI needs lots of data inputs to learn. Some people believe that for an AI to produce useful results, it needs tens of millions of data points gathered over a long period of time. We found that you can actually build a usable AI more quickly with a high level of supervised learning and applied heuristics by the internal team. Needless to say, the more data that feeds into the learning model, the better the AI gets at its tasks. So there's lots of data being generated by our system, and to get a sense of how much data we actually see and how quickly it comes, here are a couple quick points. We are sending many thousands of emails per day, and each email is generating an average of 10 analytics events per day, so multiplies out. There is also the needle in the haystack problem of classifying incoming emails replies for each sender and deciding whether they're relevant replies to the sales emails or not, because people get a lot of different types of messages, and a lot of them are just noise, so this means we must parse and make sense of every single incoming email for every single sender, and all of this combined currently nets us about 20 requests per second into our analytics engine 24 by seven, and the number is growing quickly. So there are basically two categories of data sources. So from the sender, it's the direct user of the product, and then from the recipients, which are an indirect user, because they are validating what the sender is sending or not. Now, there are four pillars of data here, and so the recipients generate feedback data about the effectiveness of the messaging coming from the blended, the humans plus the AI, inside an analytics engine, and then the senders generate three types of data, content, activities that they perform while using the product, as well as their preferences. So first, a little more on the analytics engine itself. Okay, so Sally is gonna send an email, and we first have our first decision point. The email is either delivered or not, and so we might get a bounce event, but if it is delivered, we go on to what we call the interaction zone. And there are basically four main things the user can do. So first, it's they can open the email, and then, from there, if they open it, they can actually interact with it further. They can click. They can reply, and this could either a positive or a negative reply, or maybe they actually open an attachment if somebody sent one. Now, the interesting thing is once we enter the interaction zone here, we can get a lot more information that was neither collected by the AI or that was in the CRM, because we now know about what the person, what kind of, where they are interacting with this data, so the geography, the IP address perhaps that they are using to do that, the browser that they are using, and then the digital formats. Are they on a mobile device, are they on a desktop or whatnot, so there's some more useful information that can be collected here as well. Now, so there are two types of events here, simple events and complex events. And the simple events are binary, so they either occur or do not, and we don't have to do any special processing to know whether they are relevant to the AI or not. These are the opens, the clicks, and the content opens. The complex events are the bounce and reply events, and both of these are related to incoming emails. So we actually have to parse these emails like I mentioned before, and then the AI has to be able to do classification to figure out, is it a bounce? Is it a reply, what is it a reply to? What's the sentiment on the reply, and then figure all this out and feed it back in to the engine. So the other important piece here is that every single event is timestamped. It's important to know when each event occurred and also how many times within what space of time to properly teach the AI about what constitutes engagement and when certain people are active. So for example, when Sally contacted Shirley, if we previously knew that Shirley likes to read her email at noon every day during lunch, the AI could adjust send times to make it more likely that she'd open the email, and until we do, we can get that data the first time that Shirley interacts with that. Okay, so next, sender generated content. Users generate sequences and they also allow access to the CRM data about their recipients. So the interesting part here is sequences. Now, sequences are basically comprised of a series of message templates that the sender wants the AI to turn into messages and then send to recipients on a specified interval or cadence. And then, inside each message template are several content blocks, and these are basically single sentences or paragraphs that represent a specific section of a message, okay, so what does this mean? As an example, a personalized greeting would be one content block, and a call to action might be another. The AI learns about what content blocks comprise certain messages and how well those messages work, and this feeds back into the learning model. So it can actually begin to improvise and switch up content blocks to create new and better messages and test them over time. So as an example of how a complete sequence might be used, Sally may think that a sequence of five personalized messages to Shirley that speaks to different aspects of her product offering in each one, sent one week apart, has the best chance of ultimately getting Shirley to want to take a meeting and move closer to a buying decision. And then, these sequences also have a user defined lifecycle. So there's logic that's configured here that basically, if Shirley responds after message two, you gotta stop the sequence, because you're gonna mess up the sales process you hit her with more automation. But there are lots of variance that people can teach the system that they want to do on this theme. So then there are sender generated activities, so this might be sending emails, edits to machine generated text inside emails, so this was like that blue text from the email before, when it goes to review, the user can edit it, make it her own, and then this actually feeds back in, so the AI actually is aware of edits and it learns about the tone of voice that's expected in the future. And then dynamically substituted snippets, so the other two blue areas in the message about the relevant case study and then the, the customers in the same space, those are actually, the user can tell the AI that they want something dynamic placed here, but they don't know what it is, but it has to be relevant to both. They mark this part in the message as something to be dynamically substituted. And then finally analytics queries. There is a rich way to query the analytics engines, so the user can teach the AI what's important by running specific analytics queries and how they slice and dice the data. And then finally, user preferences. So these are configuration and settings. Content preferences, so you might have noticed that when we were building the message, there are usually lots of ways that the AI could actually personalize the message and insert many different possible types of dynamic snippets, so users like certain ones for certain types of people, and so they can actually teach the AI which ones they prefer they show up more often, and then the tone of voice. You're gonna reach out to let's say a VP much different than a lower level individual contributor, let's say, and so, you can teach the AI what kind of tone of voice you would like for those different classes. It's learning that on its own, but the users can also specify the thresholds. Colloquialisms. I might call MIT Massachusetts Institute of Technology and vice versa depending on the case. So when to use these types of language constructs, and then finally, putting all of this together. So across the top here, you have the flow. The general flow, so the sender, Sally, constructs a sending sequence, recall this, this is a series of templates and messages sent on some interval, then she's going to enter her contact data, allow the AI to look at the CRM. Than the AI is gonna enrich the contact data, oh, by the way, the first two are actually feeding into the learning model. Sally is teaching the AI as she does those things. Then you'll notice the AI is using the learning model to enrich the contact data and generate the email content. And then it's gonna pass it back to Sally to review the generated emails, and what she does there is gonna actually feed back into the AI learning model. And then, finally, the AI is gonna send me a proof email. Notice it's the AI, not Sally, because the AI has some, has a lot more information on the recipients, and when the best time to send to them would be. And then at the bottom, you'll notice the recipient signals all those analytics we've talked about are also feeding into the learning model. So moving to building am AI product. Recall that before we talked about building a product more generally, we found that building an AI product is different. So once we finished our MVP and then thoroughly tested several iterations of our product, we found that the stages for building an AI product started to be slightly different than the general steps to building a product we were expecting. While the customer discovery and customer validation, components were exactly the same, I'll lead you to type of approach emerged. After the successful MVP, we found that our users were continuously bombarding us with shocking amounts of detailed feature requests for how we could make our products smarter and help them solve new products that they hadn't even thought of before they started using the AI to solve these ones. As a result to this, we have continuously had to iterate on the product rather than go to a predefined scaling approach. And something that was really important to our product development process and it's really allowed us to iterate quickly with different user feedback has been this technology stack we chose which is shown here. So I wanna key in a couple points here. We chose to use Heroku, which is a platform as a service, because we wanted to focus on the product and AI aspects rather than the infrastructure. The infrastructure is of course important. I don't mean to downplay it, but if can be outsourced, then it's easier to focus on the unique aspects of your own product. Now, we chose to build upon Google apps, because we found that most salespeople live inside their inbox all day long, and a Chrome extension is a really cool technology that's able to directly modify and extend the Google apps in Gmail inbox experience in ways that a normal website cannot. Normally, you can only run JavaScript if your pages serve from your own server, but in this case, it's actually served from Google and then we can actually come in and do things to extend their platform which is pretty cool. And then we chose to leverage software frameworks such as Ruby on Rails and AngularJS in order to speed product execution as well, instead of trying to solve lower level programming challenges that had already been sufficiently solved by open source projects for our purposes, so we leveraged those rather than reinventing the wheel. Now, since much of the raw data is messy and needs to be stored at least in intermediate states with unknown and changing column names, a.k.a. we don't know if people are gonna call on their CRM first name as name, name different things, so we might get data in there that needs to be cleaned, but also needs to be stored in that intermediate state in a way that we can predict, so it's hard to do this with a traditional relational store, so we opted to use a hybrid NoSQL relational structure, so we don't have relational components, but we also have NoSQL components in there, document stores, so to speak. So the newer versions of PostgreSQL allowed for the JSON column type, which is pretty neat and it's an effective object storage, but also feeds nicely directly to our frontend where JSON is a native object type. So this has kept us really agile since these requirements change, especially with an AI product, much of the data model is easy to adapt to new data formats, as demanded by the market, okay, and then, one of the great challenges we found is because we have a lot of interfaces, AI products tend to have very quickly changing interfaces and data models, so one of the important things we've had to do is to do everything possible to balance data storage simplicity but also with speed of access, so that that data can be gotten quickly and served up to users in a useful way in milliseconds. So some final thoughts. And in kind of in summary here. So highly personal digital interactions are the future and they are gonna become increasingly common. I think a lot of us see a lot of that already unfolding in a lot of different areas. One common one is the ads we see, but now, as you can see, emails are becoming this way too. AI augments humans, creating better cyborgs. Again, recall that these messages are not being completely written by the AI, it's taking cues from the humans, so it's basically making them better. It's giving them better information and structure and so we found that you're not gonna actually be able to remove the human from the process anytime soon. So cyborgs are a good next step. And then building an AI product requires the ability to iterate quickly off of user feedback, which differs from the traditional customer development model slightly, and when building an AI product, make sure you prepare for large, quickly changing data sets, 'cause it's always gonna be a challenge. All right, and that is all I had. My-- - [Lois] Go ahead Bryan. - [Bryan] Okay, so my email, if you have any direct questions that you don't wanna ask here, please just email me at bryan@nova.ai. I'm pretty responsive and we'll let you your questions and answer. - [Lois] Thank you very much Bryan. Now is our Q&A period, so we have our first question here which is from Hussam Dessam. He asks what kind of infrastructure would you need to deploy your solution? - [Bryan] Yes, so correct me if I'm going at this the wrong way but basically we need a high-speed database, and an elastic infrastructure. So if you're talking about like an AWS perhaps, Elastic Beanstalk, we have that, so basically the infrastructure can scale as needed, especially, talking about the analytics there. It gets really heavy and we get spikes and we need a lot of compute instances to actually be able to handle the load and especially when we are parsing the emails to be able to figure out their content and context and classify them. When the AI is doing that, that can actually get a lot of parallel environments all at once to actually be able to do it in real time manner, so did that answer it or were you asking about a different part of the infrastructure? - [Hussam] That was my question but I'll follow up with you with an email later, okay? - [Bryan] Okay, great. - [Hussam] Thank you, very informative. - [Bryan] Cool. - [Lois] Thank you, from Louis Benson, to everyone, excuse me, from Louis Benson, can you get around browser settings that prevent the outgoing data you need to do your analysis or are these just treated as a lost sale? - [Bryan] Yeah, it's a good question. When I was talking about that interactive environment, or interactive phase, right, the opens and beyond, now, we're not doing any hacks or anything like that, so there is, there are accepted ways of doing this, so it's basically a small pixel which pings our server when an HTML email is opened. It's invisible in the email, but it's like a beacon that call it. So if people want to block this, then they can. I think Gmail did a somewhat controversial update to their Gmail platform, so Google did this. They basically made all images shown by default, so you as a user have to configure your browser not to show them. Now, I don't know why they did this, but that just means that nine times out of 10, if someone's opening the email, we're gonna get a ping to our server, and but if we don't have any information like we can't get to open or it was blocked perhaps by the client, then we just, we don't have anything to work off of, so we should use it at we were unable to get any data on those recipients. - [Lois] Thank you. Folks, again, if you have any questions, please just type them right into the chat section addressed to everyone, and in the meantime, I wanted to let you folks know that the next MIT SDM Systems Thinking Webinar, it will be held on October 17th. The topic is Agile Projects Dynamics: A Strategic Project Management Approach Using Systems Dynamics for Large-Scale Software Developments. The presenter will be Firas Glaiel. He's the corporate technology area director at Raytheon's, and we normally have these webinars every two weeks during the academic year, but this year, because of the conference and related events, it will, there will be about a month gap. It looks like we don't have any additional questions, so with that, I'd like to say thank you to Bryan and to remind everyone who has attended that the PDF of this slide and a recording of the presentation will be made available on the SDM website within about a week, and you will be sent a link to the materials as well as to Bryan Pirtle's contact information. Bryan, thank you very much, really, really interesting and you are welcome back anytime. - [Bryan] Great, thanks a lot, and thank you everybody for tuning in. (upbeat music)