Industry sign banking kansas word myself
welcome to cto confessions with tc gill brought to you by it labs well hello ladies and gentlemen this episode of cto confessions is brought to you by the one and only it labs providing technology leaders with purpose-driven development teams for high performance innovation and productivity please think of us like tech leaders favorite off-the-shelf service providing quality high-performing teams off the shelf and your host today is more tc gill i.t labs chief talking officer cto and i'm speaking from the uk london a small island with a colorful track record and in this episode we're going to talk about cto's voice from the banking as a service industry and setting up high-performing resilient remote teams and architectures to talk about this we have taylor lilly cto at white label mfg so what are the key takeaways from this podcast number one how to treat your highly intelligent engineers not to get them to be tas doers but harvesting their intelligence for the best outcomes also number two communications communication is the key to making remote teams deliver the outcomes you want and thirdly how to create high volume resilient architectures and finally why as tech leaders you need a business head as well as a tech head so taylor is going to share his insights from his very very journey he's a straight talking chat and rewinding his timeline back on his experience you get to see his colorful journey one that started in the united states marine corps get that so let's not delay any further let's greet our guest from california orange county united states of america so taylor it's great to see you how are you my friend i'm doing well doing well stay and chill here in soccer excellent that's good so thank you for offering your time i know you're a extremely busy man um and uh you know and offering your time to kind of convey some of your wisdom and knowledge to the kind of tech community out there and so to start off with that you know i'd just like to kind of cover some of the some of the companies that you've worked for and you know because you've obviously got a nice a beautiful track record of interesting companies um so tell us a little bit about your kind of journey your your um career sure um i started off just doing some low-level programming way back in the day in the dallas-fort worth area which is where i you know turned myself into a software engineer and i worked for some larger because of the dallas environment i worked with some larger fortune 500 companies at frito-lay and johnson and johnson amerisource bergen the biggest company you've never heard of some things like that but pretty quickly realized that i had sort of an entrepreneurial bent in my jeans somewhere uh and so moved out to southern california partnered with a guy here built our own software consultancy back in that day which is where the relationship to it lab started um and so ever since then with one brief stint at another public company um it's been you know either like brand new startups or uh or mid-sized companies um but but not giant companies anymore um and you know the tech the fields of technology have ranged from voice over ip to uh gis and demographic analysis um then you know pay-per-click advertising and automation and that arena at local.com um and then uh and most recently sort of fintech with a heavy emphasis right now on payments right and build out our suite of services into you know a larger array of fintech oriented stuff sort of banking as a service i think i would call it all right okay that's an interesting term banking as a service bus yeah yeah and and so obviously it's quite a wide range of experiences so from a technology leader you've kind of navigated quite a lot of revolutions i guess within the within the tech space yeah and i think that really was just driven more or less accidentally or serendipitously by the fact that there were demands for those technologies at that time you know um we were eyeball deep with with a handful of guys at idt labs and building the the gis enabled um it's a really fascinating application and that company is still in business wow um uh they came out of not consulting for non-profits which is something i had never thought of but um nonprofits like uh honestly like churches but there are these uh like christian affiliated non-profit groups that do um you know sort of mission work and it might be as simple as uh soup kitchens or building houses for homeless people or whatever um and before they do those projects they do some demographic analysis and and decide because they're putting a lot of money into it yeah um and so they do an analysis to figure out where they should spend that money to do that like where do we need a kitchen just just like a an entrepreneur would who wanted to franchise a mcdonald's you know yes um that sort of thing so one of the guys came out of that one of the founders and one of the founders came out of um public school systems uh he had been the superintendent in irvine here in orange county which was um you know fairly large well-known uh reputable public school system um and the school boards i'm going to digress a little bit into this because kind of it was fascinating to me at the time the school boards have the same problem they need to do analysis to figure out you know are they overstaffed under staffed like where are new houses going to be built where is the population going to grow within the city do they need another elementary school on this side of town do they need to reduce the number of teachers at the school on the other side of town all that stuff it's you know they're they're wielding a lot of money and so they need some way to you know some data-driven way to make decisions about how to allocate budget yeah um and it had been really for years and years for decades it had been really just kludgy i'll use that technical term yeah it was like these these guys that had you know arcgis and were basically engineers of some kind not software guys would say well you know we've got these maps and we've got your city mapped and so give us 10 years of your enrollment data and just throw it over the wall to us and we'll come back a year from now and give you some idea right yeah and uh mike and dean came to us and said there's got to be a better way uh here's our vision you know they were like we want a school administrator they're still going to have to give us their enrollment data right just like how many how many kids in each school buy you know with the full address information and all that stuff we want them to give us our data and then we're just going to load it into the system and the school board can sit at a laptop in the conference room and see a map in the browser and draw arbitrary borders for school attendance zones on it and it'll just recalculate the 10-year projections of enrollment per school in the district when they drag polygons around on the map wow and being the sort of person that i was at that age and needing clients i was like yeah we could do that the perspective now you you see things like that you know now we have google maps google maps was not released when we started that project wow um google maps released after we were about nine months into it and our maps looked like crap by comparison so when i first saw google maps and how smoothly you could pan and zoom into all this stuff i was like oh my god we're screwed really like my knee-jerk reaction you know i was like we're so hosed yeah and i realized i was like wait this thing then i started playing with it i was like wait this thing doesn't do any of the stuff we're doing like it's great if you just want to find a pizza joint and drive over there and get a pizza but like it doesn't you can't turn on and off layers you can't zoom to arbitrary you know uh zoom levels you can't and you certainly can't draw poly bounce on it and affect the school enrollment projections so with that that initial like oh my god this thing looks amazing what are we gonna do you know uh but the longer the part of it is that we did it i mean we we built the version i did tons and tons of research and we went and found a library by some guys out of kansas uh that were also net developers and they had built a library that um that you know you put the shape files in and it would render the maps into the um into the browser and it had a full api so we built tools and you could literally just draw a polygon we digitized the boundaries of all the existing school enrollment boundaries yeah you know really inadmin like we made it i i prototyped the first thing and you can like remove a point from a polygon and add a point and just change the shape on the map and then click go and it would do the spatial queries in the back end pull all the demographic data and new housing starts and school enrollment history and all that stuff and feed it into the algorithm that dean had specified for us yeah and it did it and it wasn't super fast in the first iteration but they took that thing out and showed it to you know they went to like large-scale conferences for public educators yeah and joe's the first version of the people and the whole like they freaked out yeah yeah nobody had ever seen anything like that it just kind of learned that that whole concept of how to get your projections on its ear like yes the bears were blown away so i'm kind of curious as to kind of how uh i imagine that big data i mean big data has been around for a while but i imagine that was quite a challenge to kind of start working with that kind of big data and i mean how did you go about creating a strategy around that yeah it was it was tough we um it was especially tough because we were not gis savvy when we started um so you know we had to figure out how to digitize the physical boundaries because the school districts were like yeah here's our boundaries and they'd pull out a big like unfold a big piece of paper you know and had school boundaries drawn on it like a treasure map you know yeah so we had to build a tool and then they would just get like interns to sit there and literally click and draw the boundaries into the system when they onboarded a new client right and then we had to ingest all the enrollment data and we would take about 10 years of data so you know that sounds like a lot of data but we're talking about school enrollment like a number of students by modern big data standards it's actually really small data you know yes um the slog there was traditional etl like crap you know all these addresses in this database have weird special characters in them that are messing with us and things like that so right um but yeah then we then we had to bring all the you know students into the system and and geocode their their location their home addresses so that we knew where all the students lived um and then we pulled in uh us census data and uh census data from private companies that also did similar you know polling um the the consultants from the company would go talk to the city and to large real estate developers in the area and get that on new housing starts so they knew where uh new houses were going to be built right and all that all that information fed into an algorithm that dean baldwin will specified for us in an excel spreadsheet wow really i mean and i mean he and jamie both like those guys are just excel mad men it's the it's the funniest thing and i finally got mad at dean one day and he's like why is this so hard i can do this in excel i was like dude that's like it's different for one thing excel is like you know treats you very favorably when there are things wrong with your data it ignores it or somehow there's some cleanup algorithms in the background or whatever you know there's a lot of strange things going on i was like but the other problem is i'm happy for you that you can do that in excel but can an administrator at a school district in sacramento look at that data no they can't that's why you're talking to you so again you're frustrated that we're having a hard time getting your algorithm to work but it's not simple uh so uh sorry it was just it was a really fun really interesting project yeah and i guess it's interesting what you say there around uh you had the obviously kind of data kind of algorithm and thing you have to do but it's also the kind of user experience so from a from a technology perspective i mean was that was that something that was kind of quite high on your kind of priority uh list of things to solve you know yeah i mean that you know one i was just sort of and i make fun of myself now and how foolish i was and i was like yeah we could do that you know we thought we'd we'll whip something out for you you know and that turned into but you know i was calling myself a cto because it looked good on a business partner when i went and tried to get clients or whatever but i was basically a kid you know um a smart one but a kid and uh that i i learned a whole lot about technology and i learned a whole lot about running a business at the same time because i was working with entrepreneurs who were trying to get a business off the ground yes and learning about their internal processes and their sales process and what did they have to say to people and where did they have to go and the sorts of people they were talking to and what those people liked and didn't like and you know um it was a tremendous learning experience both uh in terms of technology and in terms of just uh business itself yes i mean that's quite an interesting point because um a lot of people imagine tech leaders don't tend to gonna get involved in the business i mean one of the things that we advocate a lot here um is tech leaders kind of leading a lot of kind of the business ideas and and directors of companies so in terms of i mean how did you go about learning the kind of the business ropes so to speak honestly it was trial by fire um you know i learned jamie is the ceo of the current company that i'm at and he was our other anchor tenant doing both voice over ip at that time um and i'd say 90 of what i know about running a business i learned from just being shoulder to shoulder with him for years right um because we did everything internally we built our own building system we built our own classified switching infrastructure i was on the calls with him to assess outside switching infrastructure that we might rent instead of spending our money to buy it and we did rent some at some point so we ran our own billing and our own customer service but the switching infrastructure was outside and then that hemmed us in because they couldn't innovate with us and so we made that yeah transition from the you know uh build or buy or rent you know yeah and i watched cycle happen many times and and i've advised clients that have come to me um who are in startup mode i'm like look go rent something what you're talking about building is going to be exceptionally expensive and you're going to need engineers on staff pretty much perpetually for the rest of your business life if if this is the route that you're going to go and you haven't proved your business model yet so quite honestly as much as i would love to take your money my advice to you is go rent a system that's pretty close to what you need cut your teeth prove your business model and then come to me when you actually know what you want because right now honestly you don't know what you want right so that's good that's good advice kind of um de-risk the opportunity you know uh yeah yeah yeah and they appreciate that and they do come back you know because because they didn't get you know i suppose the upside of you know you know we were laughing about how become a programmer and you'll never um never be looking for work you know yes yes that actually puts uh you know people in my position or at my level in a comfortable position that when a prospective client is coming to you or just a friend or a colleague from you know they're coming to it for advice you're not starving for work you don't have to blow smoke you know you can you can afford to tell people straight and give them solid advice even if it means that you're not going to make money right now yes but you're going to come forward that's qui
e interesting actually from uh we were talking um with it labs ceo um barney around uh integrity you know having that integrity to say what needs to be said you know yeah obviously pays off um to be able to say what needs to be said yeah absolutely and and you know i i get to shamelessly plug it labs here uh the reason that you know blacko and i are really good friends and have been friends for years and years um and the reason that i always come back and always work with it that's why i was saying earlier you know before we got started that um pretty much every company that i've worked with in the last 15 years has at some point worked with it labs um so i'm plugging the crap out of you guys chuck is in the mail right yeah but the reason is that that integrity and that trust um you know should you offshore like i don't know do you know who you're going offshore with do you know those guys yeah but you know um it's a it's a big difference that's right and i'm kind of going off uh off my set of questions here a little bit but i find it quite interesting in terms of uh you know this integrity piece and and building relationships because you hear a lot of stuff about offshoring onshoring uh outsourcing and you know cons and pros and cons around that um and so i guess the question is is that how can as tech leaders kind of de-risk that yeah so um as as always communication is super super key you know um i highly recommend to anybody that's going to work with an offshore team that if you're starting a project of of any actual size you know if you're just having somebody throw up a brochure page because you need people to find it's an electronic business card for you yeah whatever sure i'll source that anywhere you know get your cousin to do it etc yeah um but if you're starting a sizable project um i highly recommend that you know in my case we're a super small shop over here so it's it's me um i came to macedonia unfortunately i came to macedonia just as the hammer kind of dropped with the whole uh you know covet issue but um but in the past uh you know even years ago when i was working with it last heavily all the time it's the same thing i came over three or four times i think at the outset of each new large scale project that we started with somebody i did a whole bunch of requirements gathering high level requirements gathering yeah almost more at a strategic level you know really got made sure that i understood the business model that my client was attacking and what their their business objectives were yeah and and i used to write you know fairly lengthy like gettysburg address emails that would take all of that and brain dump it onto the guys in macedonia and then i would also come over and spend some time because um it's amazing what gets lost in translation everybody that i've worked with at it labs over the years has had you know solid english speaking skills but there's still something really key about being in the room with the dry erase board and waving your hands around yes um it just makes a huge huge difference yeah um i i had a a negative experience of that uh a couple of years ago i was working with a team in argentina we've been working with them for months and i'd sent emails i had calls and zoom meetings and all that stuff and i went down there but mostly i was talking to sort of my counterpart down there they were an outside company and they had their own cto mostly i was talking to him and i went down there one day and he we were about to have a meeting with the engineers and he said to me hey i'm sorry i've got to run home he had a new baby at home right he's like i'm sorry i've got to run home and help my wife out i'll be back in a couple hours if you want you can just go ahead and meet with you guys yeah so i went into the conference room i've got seven people in there and seven software engineers in there and i started talking they're all like they're all kind of nodding and i turn around i started drawing stuff on the dry erase board and i turned around and all seven of them were just like just mouths of gate they were like holy crap they had no 30 000 foot view of what the strategic objectives of this thing they were building was it was getting lost in that layer of management and they were getting tasked with millions of things to do and they were cranking out code but none of them knew where we were going you know like uh and it was evident and when i saw the looks on their faces i was like oh man yeah we're in trouble you know and i was down there because things were kind of lagging and you weren't getting the results we wanted and then i realized why because they were all you know like walking and tackling and handling minutia all day long every day nobody really knew what what the machine was but the thing was what we were building not the ultimate outcome of what you were producing was kind of missing i would admonish or caution any you know tech executive or or even at the manager level to make sure that you don't the the in delicate term is you know mushroom your software engineers these are highly intelligent people yes right that's what they're doing in this business so don't don't treat them like low-level employees and just hand them tiny tasks yes like everybody in the trench you know as from my military background the lowest rifleman needs to know what the strategic objective of the of the battle plan is yes right some of them won't get it but most of them will and they'll make better decisions yeah you know because they know why they're building the thing not just what somebody told them to do yeah that's that's a really important point uh from a tech leader perspective is is is being able to kind of communicate the vision and the outcome that you're trying to achieve because um again probably from you know you've experienced this as well is that you know you get more eyes on the ball you know if you understand what you're trying to do with the ball yeah you know they can do something yeah precisely and you you know you get different cultural responses to that you know um there's a there's a cultural thing in india over and over i hear tech leaders talk about the fact that their developers in india don't want to tell them when something's wrong um it's it's sort of ingrained in them they're just like no it's good how are you doing everything okay yes it's okay like you don't it doesn't bubble up yeah with macedonians i get a very very dry like report of what's going on uh and then occasionally you know i get it i've worked with an engineer under the i.t labs group who was just like you don't want to do that that way can they push back and and i actually appreciate that i'm not you know the sort of leader who's like uh it's got to be my way i'm like no i'm paying you guys to be experts yes so i will set the strategic objectives and i will give you my advice based on my experience as to you know how maybe we should attack solving that right but you guys are in the trenches all day long every day working with the latest and greatest technology that's out there so if there's a better way tell me yeah i love it so from a from a tech leader i'm kind of imagining that you can can design it to be this way regardless of the culture you know this is what i expect from you you know if it's not yeah i think um you know if you're working with a team or an organization that is not naturally prone to you know surface their complaints or you know surface their disagreement um you've got to build trust early on in the game um and i've often just repeatedly told you know the engineers i like to talk directly to the engineers at least on occasion um you know on this project we're currently working on i talk to the engineers every day um but um in some cases you know the team has been larger and there's a layer of project management in the middle and you know depending on what you know yeah i'm talking occasionally but i i try to make it very clear to them from the outset all the time exactly what i just said like look yes i've been doing this for a very long time and i do have opinions about how things should be handled but they're going to be high level opinions yes i quite honestly i don't care if you use angular or react you know like what are you comfortable with and what are you going to produce high quality code in a short amount of time using great i'm fairly technology agnostic in that sense yeah um it'll it'll be larger scale you know things that i'll push back on and usually it has a lot to do with like how to manipulate data as the data gets a little bit bigger yeah we'll be general details but um because a lot of the engineers that you know i work with that are at this level maybe they have five seven years or whatever and they're they're very good at uh the low level technical abilities that they have they're good with the tools they're good with ide they're good with you know this library or that library um but perhaps they haven't worked on larger systems that had to scale right so um you know you asked that guy like was there an index on that field and they're like home you know you're right i'm like yeah the database being fast isn't magic like it's a little effort into that yeah yeah so it's just things like that that quite honestly i learned about getting burned yes over the years you know yeah so yeah so it's what i'm hearing is from a kind of tech leader's perspective it's just kind of um especially around this kind of outsourcing you know when you're working with teams uh kind of out there uh outside of your you know kind of core organization is to design uh the the relationship in a way that it's not just about them delivering some output to you it's around getting them really collaborative in creating the solution that you want you know or yeah that you want yeah truly like i want them to understand i spend i would say as much of my time talking to my engineers about the business as i do about tech beautiful that i don't know if that's strictly true 50 50 but um but i do talk to them about it a lot like i tell them you know this thing is getting held up because we haven't finished the contract with the bank like they need to have a sense of what's going on in our office over here they're on the other side of the world you know um and if you just get on the call with them or send them an email at every day and say not yet yeah you know they're like this guy never tells us anything you know yeah so they you know i like for them to understand the business environment reasons why we're making the decisions that we are yeah sometimes uh sometimes we make technical decisions that go so far as sometimes you just appear to be stupid you know what i mean from pure engineers he's like why would you do that you know and i'm like because i hate it but i know it's tech debt and i know we need this thing to work this way now and i know we're gonna rip and replace it in three months or six months yeah so i know that looks dumb now but it doesn't make any business sense to spend time to do it the other way yeah right right now yeah sometimes that's hard to convince an engineer yeah that's why you know i love about what you're saying there and taylor is is that by taking a little bit of time to explain you know where you're going with something it kind of makes your life easier in a way from a tech leader perspective because people understand and they're not going to fight it they're not going to wince or have a you know the coffee room and have a quick ring at you you know about you right yeah on rare occasion you know i've made a technical decision and pushed engineers some to do something that later on i felt bad and i knew they were irritated with me because they just recoded that thing you know why did you you know um but those those decisions are pretty rare yeah from a technical perspective uh you know we'll be nerdy for a second thank god that most of the modern frameworks for building web applications are based around dependency injection now it really it really gives us a huge leg up and i think a lot of engineers that are coding now are unaware because they grew up like they're they're di natives they're dependency injection natives right so every application they've ever built was that way and they don't realize how easy that makes to rip something out of the back end and replace it with another service that does the same thing right um we we all learned that by getting burned you know by tightly coupling things back in the day and then like crap this is going to take weeks to unwind yeah so it's uh it's it's a great thing we're getting we're getting more modular more maintainable and extensible applications out of the box these days excellent because um you know net core for instance like bi is a first-class citizen it's built that way from the beginning right yeah yeah you really don't have a choice yes which is kind of good it's kind of evolved what it sounds like is that the industry's learnt from its own kind of uh you know rough rough journey which kind of brings me nicely onto the kind of architectures um architecture question that i've got here um you know what lessons have you learned around building high performing architectures as i worked on applications that that had to scale up um invariably in the applications that i've worked on the scale issue happened in the data tier um on rare occasion there was app tier code that was causing performance issues but uh and to clarify i've spent my entire career working on on business applications um right almost entirely web applications that serve a business need not consumer facing um you know high scale applications so that that's why the performance issues that i've found are bottlenecks almost invariably at the database um as the call detail records stack up or the credit card transaction records stack up or you know etc um at local.com we were pulling in multiple gigs like per text file just downloading csvs of click level advertising you know ppc data from um adwords right and uh bing ads or whatever they renamed it to um so that was a ton of etl and learning tricks to try to make that process not bogged down because we're gonna pull in multiple just gigs and gigs and gigs of data every day and that stuff needed to be you know cleaned and aggregated and broken down and stuffed into a data mart into an olap cube and all that stuff and like those are different performance issues that you have when you're just shoveling massive amounts of data around right so i'll take a specific example um or two general answers and then we'll talk about a specific example making as many things asynchronous as you can right is blocking and tackling um which these days on aws we're talking about [Music] sqs and kinesis um and well i guess those are two facets of the same thing um so a specific example in our our card transaction processing credit card transactions we're talking about here right this is sorry this is the gpg gpg right so and now we're currently in flight on projects building the second generation of that um which is going to be branded under globally paid and it's a completely just greenfield ground-up rewrite um starting with the credit card transaction processing credit card transaction processing is not rocket science right um but we need to be able to do a lot of them really fast yeah um and when you start running into we haven't quite run into the atm problem at our scale but um you don't for instance want some knucklehead out there to spin up two thousand threads and refund the same credit card transaction two thousand times simultaneously before your system figures out that they just refunded themselves fifty dollars two thousand times yeah right it's little things like that right um so there's some interesting problems to solve under the hood even though the business itself seems fairly mundane right um at any rate in gen 1 of the system way too much running back and forth to the database to get configuration that uh that guides the the business logic decisions about how to transact does the amount exceed the max ticket um have they done more than a thousand transactions on this merchant account in the last hour like rate limiting and things like th
t right um and the you know the uh the core of that logic as it stood when i got to it had been written a decade ago by somebody who wasn't very experienced and everything was very serial you know um and so it took forever to process a credit card transaction we were burning cpu cycles on the database server and the app servers and you know like everything was just kind of bogged down and i was looking at jamie going look it functions it does what it's supposed to do but it doubles your client account tomorrow if you doubled your volume tomorrow the thing would just melt you know it's really really inefficient um and so it's fundamentals that i think most experienced software engineers know if you can do things in parallel do them in parallel if you can do them async do them async if you can cue it and do it later yes do it later and one of the things that point out jamie is you know he had specified that for his like reporting and reconciliation purposes he had all these kind of tables that were logs of you know the transaction coming in the front door and the transaction going out the back door to the bank and all this stuff you know they were reporting off those same tables if a client came in and went to their dashboard and pulled a report it was pulling reports in real time off the same tables where the transaction engine was writing new transactions you know yourself yes you engineered your own contention in this application you know um so and it was just through lack of experience so um i don't know if i answered the question i kind of wandered around there yes that's cool the the thing i'm quite interested in from the kind of global payment gateway um obviously what you've described here is just the sheer volume i mean in terms of the process it's quite simple you know yeah yeah but it's it's the sheer concurrent requests that coming in and having to deal with that um but uh so having to being able to write that up from scratch i mean that must have been quite a nice opportunity to to have this being able to uh do a reiteration of of stuff that you've done before yeah the the the architecture of this application versus gen 1 is unrecognizable and it utilizes a fairly wide array of this is all on aws um so you know we're running.net core on linux boxes that's new in the last couple of years since they open sourced net we're using kinesis and key management's kms key management service um the s3 web application firewall uh route 53 uh sqs sn as like you know it's like it's all all all the big blocks they're right we're not right now we're not doing the ai stuff and we're not doing you know we don't need i can't remember what their voice their alexa thing is whatever that's called um we're not using stuff like that right now um but who knows we might in the future yeah we my clients calling and talking to an ivr you know you know if you'd like to check your balance say balance you know who knows we might be doing that stuff in the future yeah but uh but yeah this this needs to be highly highly available and it needs to be fast yeah um so we're gonna be in multiple you know have multiple app servers and multiple availability zones all tied together and then and then we have those problems to solve of of synchronicity you know wherein if if a refund i'm going to use the refund example again if a refund is hitting the load balancer in virginia yeah and saying refund because somebody's trying to mess with us is also hitting the load balancer in oregon in a different availability zone you know somewhere there needs to be a distributed mutex so that we can make sure that their their account so something needs to be locked some locking mechanism so that you can't refund the same transaction at the same time yes of some sort you know yeah exactly um so we're going to try to do that on native aws sources um i've got buddy who's a cto who's super experienced for the payment space he likes a product called hazel cast uh it's a like a um in-memory data grid uh he refers to it and it's kind of built for that sort of thing um i just haven't played with that yet but we might so a little nerd nugget there another podcast on that one that sounds yeah yeah so i i don't know that much about it oh my god jason will talk your ear off about that i've been doing payments for a long time yeah at the cto level so uh i i lean on him heavily in these sorts of matters so uh i've just um yeah i've just kind of suddenly started to understand the gravity of dealing with these kind of large amounts of data and geographically located in different you know data coming from different regions and and being able to lock accounts because we're not talking small bucks here you know we're talking huge amounts of money if something kind of goes skew with you know from a security perspective it's yeah exactly so you know um individual card transactions from consumers you know the the average ticket is probably less than 40 or something like that you know no individual transaction is very large yeah uh but as we scale up um you know we're providing interfaces we're in our clients who are you know smbs doing you know minimum 10 maybe 100 maybe 300 million dollars a year um volume and on any given day uh you know we're reflecting their corporate balances to them on their merchant account which are in the multiple millions of dollars yeah so and then you know they've got to be able to move that out to their corporate bank accounts and do things and that's where we'll really start to build out our suite of services um is we'll say well look you know like all your revenue is coming through our gateway so just let that sit there we're not a bank you know we've got sponsor banks behind us but we'll say just leave that money sitting there like you're gonna you're just gonna move it i mean if you need to move money to your corporate accounts for some reason do it but if you just need to pay a vendor in china or or you know philippines or wherever it is because you bought some t-shirts there to be printed in some other location whatever you know whatever their business is just use our interface to send the money straight here yes you get fx rates from us than you will from your bank anyway all right um you know because we're we're in it for the making the the transaction fees and the margin on the credit card processing yes uh you're going to get favorable rates on that and we're not going to charge you to move money out yeah we're not going to mark up the foreign exchange rates to move your money to another country you're going to pay a small fee and razor thin you know fx margins yeah we leave the money here we're banking the sky as we you know informally call it love you know yeah so yeah i would say quick books and we'll integrate with um i can't remember what the other big one is called my buddy was just talking about the other day some of the larger you know not sap but the big common um you know accounting as a service sort of like quickbooks online et cetera so yeah you can just wing invoices back and forth to our system and pay and get paid you know directly um that sort of stuff yeah it's kind of quite cool it's kind of solving a problem for the for the industry as well and uh you know um it uh it seems to uh well you're looking after you're looking after your kind of uh your people as well you know your end customer um yeah so the other area i'm quite curious of because they're gonna sheer quantity of information properly how do you kind of deal with kind of disaster recovery well how how how do you manage that how do you even start yeah um well the good news is that on aws we've got massive redundancy from the get go right um so you know every every dr plan has to be kind of tailored to whatever your business model is in our case um lots of the lots of the data that's in our database behind the application is really just configuration data that almost never changes right um then you've got the transactional data that's affecting balances and whatnot right so we've got tons of redundancy there that stuff is getting written to cues and then process later and that has retries and it's you know raw logs are written here and there to s3 and you know there's like that data we're actually probably creating four times as much data on disk through the cloud wow as is strictly necessary um because we've got so much redundancy in that way yeah um you know you guys better do their job because we're you know we're leaning heavily on them to uh to provide that eleven nines of above time that they talk about all the time yes um and then of course there's standard you know backing up all that transactional data out of the databases and you know you know but yeah you're right in our business you know being offline turns into a material hit to the bottom line very quickly yeah i can yeah it makes my head hurt thinking about it you know yeah yeah and and in terms of um so again kind of digging deeper into the kind of technical side of things so in terms of that redundancy i mean do you do like kind of two path redundancy or is it kind of multiple paths um yeah so you know if you want to try to picture the application architecture diagram we've got you know in the old days we called it a web form i guess you know we've got multiple sets of app servers behind load balancers in multiple availability zones um at present only in north america um we're not going to move anything to availability zones that are off the continental u.s um immediately anyway so you know we'll have east coast and west coast and as we feel it's necessary we'll add infrastructure and availability zones uh right now i don't know that the cost is really gonna work out for us uh at the moment we're gonna get started on the new serverless aurora um multi-ac clustered um so that should be kind of interesting i think that probably the just pretty much immediately as we start to load the application um that's gonna be too costly and we're just to go to a traditional rent by the hour but you know we're going to be cutting our teeth on the first low-level production volume on this system in the coming months probably yeah and i didn't want some giant multi-az cluster back they're burning dollars and my ceo is going why am i getting this giant bill every month and we're not doing any you know it's gonna lift that argument in the bud yeah well that kind of kind of segues into a nice area because one of the things you don't uh from uh you know technical leaders perspective that they obviously are experts in the in the technology side but what i'm hearing is that there's a kind of financial side to as well so the decisions you make in terms of your architecture are going to have a huge impact on that yeah well there's so many tiny things to look out for uh you know when you're in the cloud the way things are built um you know everybody is kind of hyper focused on uh just your hourly rate on your ec2 instances for instance right because that's a that's a big line item depending on your application and how it functions that's probably the meat of your bill every month um but if you're doing super high transaction volume as we intend to do and you're using something like dynamodb that's costing you for i forget how they're doing it now like per bytes that you're moving in and out of dynamodb tables um [Music] you could just have an inefficient sort of i don't know what to call that there's an inefficient implementation in the way that you use dynamodb and you could wait for a month and just be like holy crap what are we doing like you know and you'll very quickly feel the pain and go why are we that data never changes why are we not cashing that yeah you know um so or or why are we making three calls to dynamodb when we should simplify the data model that we're storing there and make one call beautiful you know yeah that that sort of thing right and in terms of being able to kind of see the kind of visualize where the cost is going how how do you do that because i mean imagine there's lots of points where that cost builds up as you mentioned here you know three calls yeah so you know aws has done a pretty good job and honestly this will be the first time in a while that i've run an application of this size on aws and i noticed uh recently that a lot of the things that you used to have to go to a third-party tool to kind of mine your own billing data out of aws is now built directly into aws oh wow those third parties are probably unhappy about that maybe they got acquired i don't know yeah but you know uh aws is reasonably quickly in my estimation been adding better tools so that you can drill into your bill on aws and see you know where where am i burning money yes so uh yeah you know it's funny and that's what you know if you think of traditional architects you know um like building architects and everybody likes to think about the the artifice the beauty of the building that they've been to built the things that you see on the surface but architects are trained in materials and construction and you know they have to they have to think about all those things um and that's that's where we end up here too it's not just how it looks on the surface but the ceo cares how much it costs to run this thing every month you know hopefully in our business you know we'll make it a truth that you know the aws bill is just going to be a funny little footnote to us you know but uh which it should turn out that way sure i'm always telling you that i'm like even even the engineer's salaries in the end you're gonna be like it's not even gonna register right uh if we if we do this right so and and and so it's the advice to the tech leaders out there in terms of having this kind of financial um aspect of your role you know because a lot of stuff is ending up in the cloud um would you say that technology leaders actually kind of embrace that wholeheartedly or is it something that people are learning the hard way uh which the the tech leaders out there so for example obviously you're switched on when it comes to the cost of certain aspects of your architecture you know yeah yeah yeah um yeah again you know unfortunately that comes from having been burned you know right uh jason the other cto that i was referring to he was just kind of messing around he doesn't use aws he's the sort of nerd that wants to build his own bsd kernels from scratch you know um but he was just trying to prototype something and i think he turned on their uh hard hardware hms hardware management service something like that it's like a a hardware module that handles encryption and decryption behind the key management server i forget what three acronym is well those things operate as one of those like pay per hour cluster i believe he turned one of those things on just playing with it one day and walked away and like a few days later realized he had like a three thousand dollar bill on aws yeah something he's just poking at you know he's not not cloud savvy he's never operated on that sort of infrastructure he must buy his own hardware and rack it up himself all that stuff you know yeah and you know his application is going to get to a level of scale wherein he's probably going to be glad that he went that direction yeah but he has not worked in startup environments or in tight budget environments before he's worked at larger companies and so he doesn't have that perspective that i have where and i'm like dude you need to prototype this thing because you haven't you haven't proved your model yet yeah like do it in the cloud do it fast do it inexpensively make sure you can actually do the business you want to do yeah and then you can start ripping and replacing layers and build bindings on hardware and stuff if you want to you know yes yeah he's kind of waterfalling it even though he's on a on a startup budget right yeah that's cool yes i hope you swim before you speak cool um so just kind of uh kind of mindful of the time that we have together and um another area that i want to ask you about from previous conversations you know it's around around teams and and how how you make them high-performing teams i mean what what would be your advice to kind of for the tech leaders out there on your experienc
yeah i think i think we touched on some of that stuff topically earlier um and my primary advice that first advice is what we talked about earlier is make sure that your the the grunts in the trenches know the battle plan right um work with the company that you trust you know because you don't want to burn time and dollars working with some guys that are bs'ing you you know oh yeah we'll buy that tomorrow yeah we'll have that tomorrow tomorrow turns into three months and then what they hand you is nothing yeah um you know i'm not i'm not eaten up with all of the pageantry of agile project management um in the sense of everything having to be done by the letter of the manuals but you know it's funny i look at what we're doing with scrum these days and it's pretty much what we fell into naturally wow 15 years ago um the word agile was out there it wasn't the buzzword bingo word that it is now but it just agile project management processes really just reflect the reality of how your project is going to end up getting managed anyway yes like you're going to wind up with the the business owner you know the stakeholder um coming to you on a daily basis and changing things or wanting to see what you've done and wanting to steer it right yes um so you know there's there are two facets i suppose to getting your project to be as efficient as it possibly can be one is making the engineers efficient you know at their desk with their hardware with their ide you know making sure they've got everything they need and they're not blocked and that they're getting fed requirements yeah um because over and over and over again every engineer you've ever talked to or i've ever talked to you asking what's wrong with the project like we're not getting good requirements fast enough yes every time yes because the engineers know how to write code they got to know what to build yeah so and you guys can go ask angela and she'll laugh and she'll be like oh yeah taylor was not getting us requirements fast enough she probably she might still say that today yeah you know but i always explain to them i'm like look i'm stuck in this banking industry thing over here like everything all the business requirements are always held up by the business because there's always something that hasn't happened yet some meeting that hasn't happened yet and so we're doing the best we can to build all the infrastructure that we know we're going to need and make it modular so that we can pivot if something mild changes at the end i mean we know where we're going but there's details in there that we're not sure of yet yeah so make make sure you have open communication make sure that your engineers are allowed and in fact incentivized to tell you when things are wrong cool um you know uh it's the it what was it toyota that figured that out like every every employee in a toyota manufacturing facility it is has the right to hit the big red button and stop the conveyor belt yes yeah that's that's what you want you know um you don't want to build a negative feedback loop wherein your engineers won't tell you that things are not designed properly or that something missed or that something's broken because they're afraid they're going to get hit with a stick yeah like that's not no let me know let me know early let's fix it and let's move forward you know so from from your from your perspective then i'm kind of curious how did you learn that obviously i mean it sounds really obvious as you describe it that to allow people to do this but you don't always see this industry um you know i don't often tell people that i learned anything in the military um it might seem like an odd parallel uh but many many years ago for four years i was in the marines and one of the things that struck me early on in my training now this didn't always strictly happen in reality but doctrine was uh you know the metaphor that i used earlier that every rifleman should know the battle plan um and it's really important uh especially when you know i piss off some prior military but let's just say that the main iq in the trenches is not comparable to the mean iq in a software engineering facility right um you're you're dealing with highly intelligent highly trained people um and so to treat them like they shouldn't or can't understand the strategic objective is a disservice to the whole organization yeah um so let everybody know this is where we're going this is the desire to end goal right everything that we're doing and every decision we make all day long should service that yeah objective um because we've got all day long every day to nitpick tiny technical decisions right but let's not lose sight of you know we're the bigger yeah yeah and how how often do you kind of reiterate that vision because i'm a big believer in envisions and and repeating them to the point where it becomes almost ridiculous because you need to remind people yeah i i probably don't repeat them often enough um you know every every once in a while it's good to just have a high level talk you know um like the team that i'm working with right now is only two engineers and a few engineers fingers on the keyboards writing code um and you know a devops engineer helping glue everything together all the ci cd pipeline and all that stuff yeah i think we're about to add a third and over time it may grow a little bit as we start to like really kind of work in parallel on a wider suite of services for our clients um but uh it would the point being the team is small and it would be easy for me to just you know once a week when we're having our our daily call anyway just sort of reiterate don't forget this is where we're going you know um i think uh i think they have a fairly good vision um i did come over there a few months ago and you know we kind of talked through all that stuff yeah um we're finally about to start passing to production traffic here in the next few weeks so i know they're excited and i'm excited because we want to kind of see this want to push this thing out of the nest yeah see it take some traffic so that's good so it's been a really interesting conversation taylor i mean you know obviously highly experienced and you work on some pretty cool stuff um so in terms of uh our audience tech leaders and aspiring tech leaders out there what would be the main takeaway that you kind of like to leave them with for this conversation um well we ranged over a bunch of topics um but i i think in terms of working with with offshore companies or teams at all actually um would be again that that shared vision and strategy on each of your projects um you know every application and every project has its own technical hurdles that you're going to have to um address um and those are going to be you know very specific quite often to whatever it is you're trying to accomplish but the common thread across all the technologies and all the projects that i've worked with is getting the most out of my engineers by making sure that they know what it is that we're actually trying to accomplish not just leave this file uploaded to this server and then pushed over there yeah why what we're going to do with that yeah so keep keep hammering that home and also uh just a sort of a general adage for me is like don't don't use technology for technology's sake uh it's easy to let your engineers go off on what we used to jokingly refer to as bbd or border boredom driven development right every 25 year old who's got more iq that knows what to do with wants to grab the latest tool yeah and the coolest new open source library and try to do something with it and i know it makes me sound like a grumpy old man but i'm just why are you doing that there are perfectly good tried and true tools that do this mundane thing that we're trying to do we don't need all that stuff let's let's leverage the open source community and and those sorts of things when we actually have a problem that we're trying to solve yeah that that that library or that architecture whatever was intended to solve sure yes yes let's not use it for it's unsafe because now that's right i i have to confess i've worked on projects where uh software engineers uh engineers and generally we're very curious kind of creatures and and we want to find new cool things to do as you say what did you call again boredom driven development [Laughter] yeah because i worked on a project once where the uh they were producing this product embedded product and they'd written it all in c and then c plus plus kind of came out and i was shocked they stopped everything and then rewrote it all again in c plus plus yeah and why well it was kind of pretty cool but um you know even then i thought something's not right here you know but yeah excellent so thank you taylor it's been really really uh you know interesting and i've learned a lot of stuff hopefully our audience has and and hopefully we'll have you on again on on another podcast to explore maybe that cool subject you were talking about earlier on let's do that all right thank you very much taylor well it was great speaking to taylor it's a pity we didn't start recording the podcast earlier actually me and taylor had a wonderful conversation with some real straight to the point nuggets of wisdom in any case what we did manage to capture in the recording proper was more than enough don't you think i like how taylor covered the subject of remote teams unless you've been stuck under a rock in the last six months remote teams have become a big thing in many cases the only thing and getting the most out of them is an art i like the insights that taylor shared here well if there's anything that struck your interest here and you want to get in touch with taylor you can use the contact details provided on this page and remember to subscribe to cto confessions podcast and it labs newsletter where you get to get regular tech articles and invites to the itlabs webinar series urls to do that can be found on this page as well we are consistently creating material to create support and nurture a tech community of tech leaders and of course if you want to know more about it labs and the services that we provide including our teams as a service service please don't hesitate to get in touch as mentioned in the intro please think of us like tech leaders favorite off-the-shelf service providing quality high-performing teams off the shelf with a wide breadth of skill and knowledge well that's all folks look after yourselves and keep safe wishing you all a good day or evening wherever you are in the world from everyone here at it labs live well and prosper
