Send Collector Zip Code with airSlate SignNow

Send collector zip code

hello everyone and welcome to our session open telemetry collector telemetry built into all software thanks for joining us first off like to introduce ourselves my name is steve flanders i'm a director of engineering at splunk i'm also an open telemetry collector approver been working in the observability and monitoring space for the last decade and was actually contributing to the open census project prior to it joining with open tracing to form open telemetry and joining me is trask hey i'm trask stolenker i'm a software engineer at microsoft uh i'm a maintainer on the open telemetry java instrumentation project and i've also been in the uh monitoring business sort of well open source business at least for a decade uh um i wrote and maintained uh open source uh java monitoring tool called glow root and uh enjoy that space so much that uh here i am in open telemetry world now awesome so our agenda for today i want to give you a quick introduction to what the open symmetry project is talk a bit about the open symmetry collector and how it fits into the observability picture then do the same thing around auto instrumentation focusing primarily on java that we will touch on other languages as well and then really wrap it up with a demo showing you how to really get started with the open telemetry project so with that what is open telemetry open symmetry is a cncf project and it's really looking to solve a pretty big pain point in the observability space if you've heard of observability you've probably heard of the three pillars of observability being the different data sources that you have in your environment traces metrics and logs well for each of these different data sources there are actually different layers to it you have the api you have the implementation of that api all the data collection aspects and then things like wire formats interrupt formats and the like some of these things are actually language specific so when you start talking about instrumentation apis they have to be written for each individual language where the agent and collector mechanisms have different deployment methods depending on the type of architecture that you're you're deployed into open symmetry is really looking to solve all of these aspects the idea is everything that you deploy within your environment to get telemetry data out open symmetry is providing a completely vendor agnostic solution too to get that telemetry data to the back end or back ends of your choice open symmetry actually does not offer a back end though that's where it kind of draws the line anything instrumentation data collection even things around semantic conventions are all included and then you can choose either an open source or commercial backend to send your data to now the open symmetry project is currently in beta for most of the languages there are a few that haven't reached beta yet there's been an extensive focus on tracing to start so it's quite mature in fact the specification has already reached its first stable release the metrics one is actively underway and that should be wrapping up later on this year it will also be stable by the time we get to ga and ga is planned for later this year so a lot of active contributions going on now and again this is really the joining of two other projects open census and open telemetry now another interesting fact for you open symmetry is extremely active in fact it is the second most active project in cncf behind only kubernetes according to the cncf dev stats so let's talk about how you would deploy open symmetry into your environment so if we think about having one or more hosts each running one or more applications and maybe we want to send the data to one or more different back ends the typical recommended approach would be to go ahead and deploy the open telemetry collector as an agent on each of your hosts that will allow you to actually collect host metrics automatically and also provides a way to collect tracing information now that tracing information as well as application metrics can come out of open symmetry client libraries these are going to be language specific and out of the box are actually configured to send directly to the open symmetry collector so if you deploy it without configuration you'll actually have telemetry data being emitted for the auto instrumentation libraries which is pretty cool and then in addition you can also deploy the open symmetry collector as a standalone service if you want to handle more advanced use cases maybe around protecting from a security perspective limiting the number of egress points or controlling your api tokens or if you want to do more advanced functionality such as till-based sampling in the case of apm now if you look at the open symmetry collector as a single component that exists it's actually one binary that can be deployed in a variety of different form factors depending on your needs it's really looking to make it easy and provide a vendor agnostic way to collect this data so it actually receives many open standards today like jaeger prometheus and zipkin and it can export to both open source and commercial backends so in a way it handles translation for you it so it understands the formats it's receiving and how to send that out in a different format in addition it can be used to manipulate the data before it gets sent out maybe you want to do like pii reduction or you want to do metadata enrichment all this is made possible through the collector as well now why would you want to actually deploy a collector like why not just have the client library send this data directly to the back end the idea is really to offload the responsibility from the application for two primary reasons one is you don't want to have to implement it in every single language like trying to keep that consistent is really hard and even the implementations are going to change because they're language specific second you really don't want to introduce too much overhead to the application so you want to do the minimum necessary to gather the data that you need that's why we recommend deploying the open symmetry collector as an agent you can actually have the instrumentation pass off this data quickly and then have the agent handle this in a unified way and be able to handle things like buffering and retry compression encryption all the things that you care about that could be additional overhead to your actual application so really the idea here is time to value make it very very simple offloading of responsibilities and really providing a consistent experience end-to-end now if we think about the architecture of the collector basically it has a notion of receivers how you get data in this is both push and pull based as i mentioned it already supports many of the open source standards including jager and prometheus it has a notion of exporters which is how you get data out and again it supports both open source as well as vendor commercial specific backends it has a notion of processors which is a way of manipulating the data as it passes through the collector so this handles things like batching and cued retry but as i mentioned also pii redaction tail-based sampling and other specific scenarios now what you end up doing is actually defining what we call pipelines in the collector so for example i can say that i'm receiving traffic over otop we've been going through a batch and cued retry processor and i'm exporting in jager as i mentioned it handles that translation of data format for you automatically you could then define a second pipeline where again maybe i'm receiving from otop which is open telemetry's format that it includes by default that's what the auto instrumentation and other client libraries follow today and i could have that then export to two different destinations otlp and in this case prometheus so there's a lot of flexibility in choice via configuration what's really nice about this is you don't have to actually go make code modifications to take advantage of these capabilities it all happens in the collector itself there's also a notion of extensions it's just a way of typically doing health tech health checks and profiling but it's also a way of handling things that might be outside typical data flows for the telemetry data that you're collecting now the collector actually has what's known as the core repository as well as a contrib repository in core you have all the open source aspects and as you can see here it's pretty broad support for both traces and metrics in the case of open telemetry logging is just starting to pick up so eventually logging will be added as well to both client libraries and the collector but it's still early for that the expectation is that they'll probably be a beta later this year for it in the case of contrib these are more community-based so this could be vendor-specific logic or it could even be open source software that only applies to a smaller subset of the the user base of open telemetry but it's a way of adding things that are outside the core functionality of open telemetry as a whole again you can see pretty rich adoption here in fact in general two of the three major cloud providers aws uh sorry azure and jcp are already instrumenting the backend services aws is just getting involved right now as well with the project all the major vendors are also contributing at least exporters if not other aspects to the project and you're also seeing broad adoption and support from many end users as well with that i'd like to turn it over to trash and he can tell you a little bit more about java auto instrumentation sorry was muted uh all right so uh the goal of auto instrumentation is to give you as much insight into your production system as possible with the minimum amount of effort so is to get you up and going and starting to see what's happening in your app in the case of java and to enable the open telemetry java auto instrumentation you just need to add this java agent command line arg to for example all of your microservices and maybe an environment variable to tell them where to send the data to as steve mentioned before by default it will send to the agent the collector agent that's running loc if you're running that locally and just with that minimum amount of effort you get this beautiful distributed trace flow picture in your back end monitoring system in this case it's a screenshot from jager which is another cncf project and so yeah so you get a lot of visibility so i did want to mention what we're doing in the open telemetry uh java auto instrumentation project is not new um the java ecosystem has a rich history of codeless monitoring solutions we've had commercial monitoring tools doing codeless monitoring for a long time we have open source monitoring systems doing codeless monitoring for at least 10 years and more recently some commercial monitoring tools have been open sourcing their java agents which has been really great because uh i like to have access to all the source code of any library or agent that's running in my production system and i imagine i'm not super unique in that way and and a special call out here to datadog who donated their open source java agent to open telemetry as the starting point for our work in this community and lastly i'd be uh remiss not to mention a couple of open source libraries that make this uh auto instrumentation bytecode instrumentation possible in the java world um asm a low level bytecode instrumentation library that's been around forever and uh bite buddy a higher level bytecode instrumentation library built on top of asm that has been around the last few years and been widely adopted by most of these types of auto instrumentation projects so that brings us to uh open telemetry the open telemetry java auto instrumentation project and um [Music] it is really i see it as a community dedicated to building and maintaining vendor neutral java instrumentation and also another important aspect of the work is working across working with other open telemetry communities other language communities to provide consistent telemetry across uh different programming languages um so in that way i i think is sort of the unique the new thing that the open telemetry communities are bringing to this evolution of java auto instrumentation and while we've been focused on tracing so far definitely we are working towards having metric support and then as steve mentioned in the future also logging capture with the same infrastructure um so this is the list of um all of the instrumentation all of the java libraries that we have instrumentation for so far and so with auto instrumentation um it's uh important to so if you are using these particular libraries we will instrument those at runtime via the java agent and capture the calls the requests coming into those libraries the calls going out of those libraries what your jdbc sql calls look like and so this is where we really think that the the power of the community coming together can grow this list over time supporting you know as new versions come out supporting new versions so the more people we have the more vendors the more community members who are all working on this common auto instrumentation um the more uh support that java the java users will get automatically out of the box so one question you might be asking is you know auto instrumentation is great but what if i want custom telemetry also um and so i like to recommend starting with just auto instrumentation out of the box try it out that way you can start uh you know seeing telemetry flow to your back-end monitoring system and then you can start to layer in custom telemetry on top of that or inside of that so depending on the complexity of your needs there's a couple of different ways to add um add custom telemetry into your application um as well as i mean the the for the open telemetry api is a full-blown telemetry api and the auto instrumentation um auto correlates captures and autocorrelates everything that's coming from the open telemetry api so those two projects uh uh blend really nicely together and uh for sure more auto instrumentation customization hooks coming for as we run into you know as community members bring new use cases things that um so we look forward to growing this project and um uh yeah hearing from everybody um so i wanted to end here with um uh i think my favorite thing about uh the uh or at least one of my favorite things about this project is that you can take all of this great auto instrumentation and and your custom telemetry too um to the back end monitoring system of your choice um so this way the this community is really focused like uh steve mentioned on the gathering side not the back end monitoring side so we can really focus on having uh really great auto instrumentation really great library instrumentation and then the back end monitoring sys systems vendors open source solutions um just need to provide their exporter for open telemetry and that can either be directly for example on the java side you can have a java exporter that sends directly to your back-end monitoring system or if you're going through the collector that steve talked about the back-end monitoring systems they only have to write their exporter once for the collector and then the java auto instrumentation flows um to that back-end system same with other language instrumentations and uh speaking of other languages um so auto instrumentation is um um growing um we've got a javascript community python community ruby community and uh dotnet as just getting going so um this is a the instrumentation is a great way to get involved in the community because um you know you can take your favorite language your favorite library and go and see how you can uh contribute uh instrumentation for that particular library if your favorite library and language is already uh instrumented go and test that out see if it works the way you would expect as an expert in that library and let us know if we got something wrong so again that's a great way to get get started in the community cool and next up we will do a quick demo to kind of show you how to get started with both java auto instrumentation as well as the open symmetry collector so to kick us off here i'm actually going to start up a docker compose file and while it is booting up i'll walk you through exactly what it's doing so what i've done is i've taken the spring pet clinic micro services repository and it actually contains a docker compose file made up of multiple java microservices is kind of outlined in this architecture diagram that you see here so it's going to spin up a variety of different containers and what we've done is actually modified it to add open symmetry java auto instrumentation into it have it sent to an open symmetry collector and have it forward that data to a zip conserver so we can actually see it on the other end so from a diff perspective basically what we did is add a collector.yaml file so you actually configure the opencylindry collector through yaml and here you'll see that we have configured a receiver basically the otop1 which is the default one that java auto instrumentation will send data out as so we're just receiving over otop and then we are going to export to a zipkin endpoint that is a different docker file that is running inside of this microservices based architecture so that's just a one-time config for the collector again if i want to add different processors i can go ahead and do that here in this case it's just a pretty standard configuration next up we went through the docker compose file and as trust mentioned you basically just add this java agent flag so you have to go ahead and tell it where the jar is for the open symmetry java agent and then uh specify what the service name is so we also set an environmental variable so for each of the docker containers we added that java agent flag and we added the environmental variable so that we'd see these as different java services in our waterfall view or service graph depending on the backend that we are using so you can see that for all the different docker containers and we added the open symmetry collector as another docker container that gets fired up into this docker compose it uses the collector.yaml file that we just showed at the beginning and the only final thing is that all the docker cont containers that you're seeing in the docker compose actually get generated through this docker file at the end where we have to tell it to go grab the open symmetry java agent jar based on the latest version that's available in the open telemetry repository today so that's really the only changes here you'll notice no java changes no recompiling of the code we've just pulled down this jar specified the java agent set the service through an environmental variable and deployed the open symmetry collector and the net result should be that we see telemetry data coming through our system so let's go ahead and see if that's happening so we are in a zipkin here we'll go ahead and look at the service name and you can see that services are starting to appear already we can go ahead and look for individual traces and you can see the traces are starting to come into the system so here we have a discovery server or service that's calling a config service you can see that this is actually using the open telemetry sdk and we actually get some semantic conventions and metadata automatically uh provided for us so you can see this is restful services we can see the status code we can see which methods being used so again some pretty rich information if there were errors like 400 500 level errors they would also show up here and as the app gets hit more you'll actually see that a variety of additional micro services will start to show up as you can see now we actually have several calls coming into the system and so each individual request no matter which microservices it hits it'll go ahead and generate trace data for that so again very simple to get started you can take basically any java application that's using one of the supported libraries that trash showed earlier go ahead and add the the jar and collector and you can go ahead and send this to your back end or you can even log it locally and just see what data is being generated it's pretty easy to extend this like if you want to add additional metadata you can do that through environmental variables in the auto instrumentation itself or as a configuration addition to the open telemetry collector quick and easy yeah so um uh we're steve and i were both super excited to share uh the open telemetry work that's uh going on with the folks with new folks um and we would love to have you join the uh join the conversation join in the project whether that's as a testing it and providing feedback uh uh providing code um there's a couple ways to get started there's a couple of gitter channels um if you post there you'll find a bunch of a bunch of people monitor those bunch of friendly folks join a sig if you go to this link you can see find your again find your favorite programming language or your favorite topic like logging or metrics and there's weekly sig meetings and there's um and uh yeah and like i mentioned earlier instrumentation is a a great way to get started i think uh biased um uh you know again uh take your favorite language take your favorite libraries go test it out if it's there and provide feedback or if it's not there um ask for help on how to get started and we'd love to have we'd love to see you and that wraps up our talk thanks so much for joining us and we hope to see you involved in the open telemetry project soon or trying it out and providing feedback thanks a lot