Create Initial Annex with airSlate SignNow

Add initials annex

hello everyone so today we are still three online so we have radovan richard and myself and we'll talk about a subject i have no idea about which is hitalex and i'm really looking forward about uh learning about it and then about the cluster etiquette at least i can say a lot but user interesting okay so hello everybody hi yeah hello so get annex what are your first thoughts when you hear it what have you heard about it before nothing so what i've heard about so i'm also as same as anna i've never used it before although i use git daily since many years and what i heard about it it's good if i want to track large files and maybe files i don't want to share yeah so i think the way it works is that it tracks the metadata of the of the files but then they the files itself can reside outside of the git repository yeah okay so what are the most important things about git that makes it so popular simple yeah visual control decentralized distributed yeah i think that's really it so decentralized and distributed so there's some other things you might often hear about so there's git lfs so git lfs is like the centralized version control system so there's one server where all the data resides um git nx is decentralized and you can have multiple different places data's stored what about get lfs i mean you still you still can decide where that central place is right yes like there are only one central place i think you can decide where the central place is yeah and next to you will have data the same data in different places you'll see so basically you add remotes to the repository so the data can be distributed among these different remotes and you can say send this data to this remote or send my data to every remote or so that one of the important things about it is when you have a huge amount of data you can't manage it in terms of copying the data or moving it or something like that you have to say i want two copies of my data at all times and you have to rely on the system to deal with that or i want to get my data here figure out where it is and tell me and then bring it here things like that and that's um why git annex can be so good for big research data so should we get started before we get started maybe we can remind again that we have this hackmd oh yes where you can ask questions and give comments and answer questions and for those of you should we put it again in the somewhere yeah i don't know how the twitch chat works whether like if you connect you see the past of the chat or oh no yeah okay so we can put it there again and then um as a reminder for those who will join later so today we talk about git annex and later if we have time we'll talk about cluster etiquette like how to what to do when on a super computer and what to avoid avoid yes okay it's a chat i just lost a window okay so here we go i'll share my screen so here's my terminal we start by making a git repository uh get i'll call it git ga demo so we start off by running git init as usual and then we'll run git annex init and let's give this a name so we'll call it rkdarst computer because one of the things with managing data is you have to be able to descriptively identify where things are so there we go so i've just told this git repository that i want to be able to store data along with it so let's make some data and can i ask so what so get in it created or dot git but what did kid annex in it create where did it so it wrote something into dot git alright so if we look at dot git we see an annex here and this will have some files but we'll come look at these a little bit more later um here we so you can do a git and explain it many times or only once so you would do yeah you would do it for each repository or each connection of the repository yeah okay so i made a file called file1 so if i get add it will add it to regular git so instead i will get annex add file one and let's look at this so we see it's blue here actually i hope that's visible in the font uh but we see it's a symbolic link and it points to this directory here so to get annex objects some directory hashes and then this big thing which is a hash of the contents of the file so that's basically the primary idea here so git annex objects this is where all of our data will be stored and in git the only thing that's stored is the symbolic link to it so let's do git status and it says there's a new file okay so let's get committed and give it our standard message of initial commit so well let's look again we still see the sim link if we do get log we see initial commit if we look at what changed in the log we see this added a new symbolic link so yeah the either the one or the two here means it's a symbolic link but it has this link to the directory or to the contents so just like git git annex uses a hash of the content in order to store all the data and that's the key that's distributed all around okay so what can we do uh let's cap the file uh objects it's the only file here and we see there's the test data inside of it so like expected let's do git annex list so this lists all of the files vertically and the rows are the different remotes we have so there's a web and a bittorrent remote which are sort of automatically added as a peer-to-peer way to distribute things but i've never used them and it has no effect unless you actually enable them or request to store data there i think in particular bittorrent was made to be a way to share data peer-to-peer without having to make your own tunnels or something but okay let's see what's next on the list any questions so far so here's a file is if you you still haven't committed the file so the file is never committed the file itself is the data of the file is never stored in the repository but the metadata basically the date it was made and the hash of the contents are in the git history so in the git commit i could say anything i wanted to about it i could say yeah this is going to be a um [Music] this will be i made this file on this date and it's what it means and so on so let's go on and do a few more things so let's make a data2 file and let's make a large file so here this makes a 100 megabyte random file that's what you call a large file but is it mean to be used for very large files or like gigabytes of kinds of things but i mean already a 100 megabyte file i would i think that's too large for a normal git repository as a file yeah it would be like an image and this is right okay so how about file 2 do you want to commit it to regular git so here git add file to and we list and we see file 2 has not been made the sim link [Music] yep okay so let's uh if we get list files it looks like expected get annex list and we see the two annexed files and these so you get ls files it's to list the file yeah is it your this is i don't know this ls fine that's a standard git command that lists all the files git knows about i just do a less yeah well [Music] here we can verify that everything that's good add it to it so these sim links themselves are really interesting thing so let's try to edit the file notice it is read only so in emacs these two percent size signs mean read only so the file is locked so why is that so if it's in the annex that means um it wants to keep it safe so before you can edit it you have to unlock it which makes another copy so basically you can't edit the only copy of data that is annexed so to me if i had a research group on a cluster with a lot of data i would be happy to use git annex even for this we make the repository we collect all of our data from different sources when a data set comes in i will annex it all and then make a commit and then in that commit i will describe where it came from and who did it and things like that you know all the normal metadata you get in a git commit and then commit it and now i can be sure that no one will be editing the files unless they're unlocked and then a change would have to be committed again which i think is really useful so the groups i've been in have always had this large amount of data that just sits there and you might makes make it read only but that can always be changed so notice in these symbolic links it stores it in a read only directory and the read only file inside of there so some of the more clever programs can't try to change the permissions on the sim link and then do that and then edit it without you knowing so there is a question on hmd about this link and i have i had a bit of a similar question so when you get annex edit the file you created assembling out of it yeah so one question on i can d what does that really mean for general use so what is the same link i think let's see right and what what i wanted also to know is that so just by getting annex adding it it does something with my file it moves it out of if it moves it to a different place right right yeah so a symbolic someone else want to explain what a sim link is or should i really that's a hard question so it's a way that the operating system whenever you try to access this file file one it directs you seamlessly to this other file and on disk here all it's storing is the reference to here and as far as any normal program knows then this is just when you access this you get this data yeah it can also be useful so i use it you know outside of gitanox it can be useful if you need the same file in different places but if you change it you don't want you want to change all of them in one all of them at the same time and then you can create the file only once and all these different places can link to that one place and then if i change the one all of them change right yeah that's how i use it too yeah um let's see i just have another question the file that has been moved to this uh dot kit that means it's not the real file is not there anymore right yeah so part of one of the design principles of git annex is that if you take find this repository a hundred years from now and git is dead get annexes dead like all these projects aren't installable anymore the information on the file system is enough to recover the data so as long as symbolic links can still be read which basically will be the case if you can access the file system then you'll be able to get your data out of it and pull it out of the annex without any special tools which is a very important philosophy yeah yes and there's even a page that in the git annex instructions that explain why this is the case like all the ways they future proof it okay so this is one repository should we make another one and start connecting them yes what do you see i'm really curious about this okay so i'm going to make a git annex demo 2 by cloning this okay so if we look in here what do we see so we see file two which has the data within it but these are broken symbolic links because this repository doesn't have it yet so if we look at the size of this repository itself it only takes up 300 kilobytes which i guess is the standard size of the git repo okay so now we will let's see have i prepared this yet so i'm gonna get annex initness and call it other i'm actually not sure if we need to do this but let's do it okay so you would say if you clone this repository and of course in the git repository itself there's nothing really special here the repository itself could be shared on github without any of the data within it okay so let's do a git annex list hmm interesting so we see here which is this copy of the repository and then we see origin and we see that origin has the two different copies and here we don't have them so what if i want to copy here which one should i take maybe start with file one that's a small one yes yeah and if we ls now we see the sim link is alive now and if we do get annex list we see that file one is now in both places so that is it's a local copy yeah now it's a local copy so how would this work let's say you had all your data on the cluster or whatever your file server is and it get annex repository you can on your own computer clone the git repository and get all the metadata but not the large data and then look at it look at the history all of that and when he needs something you do get annex git and you tell the file name yeah that's really neat yeah but the the file is always on the local computer or can i put can i get them remotely well because here how do you specify where the file is really physically located and related question from imdb is so where is the where can i store it can i yeah exactly location b can it be a cloud where is it typically yeah so the first well maybe the first question is where is it for this repository well for this repository as far as i know it's always like when a repository a git repository has a copy of the file it's always in that repository itself that just sort of how it works i mean what does it mean to have access to a file if it's not available somewhere um yeah it's like you can't use the file on this computer if it's not put on this computer somewhere and if it's put on the computer then it's put in this repository it's a get is a like a copy right yeah yeah but can you somehow store the files in a cloud or did i answer the question so that's what we'll get to we'll get to there next yeah uh so far we're doing local remote so hmm no okay so if i do get annex info i see a list of all of the different remotes that there are so we see there's rkdars computer and other so these are both regular remotes which are actual git repositories what we'll do shortly is special remotes which are not git repositories it can be an s3 bucket or a file system or basically any kind of key value store can be used to actually store the data and that's pretty neat okay but let me see what else was going to be in this section ah yeah so now i'll do git annex sync and part of what git annex does is distribute the metadata around so it assumes that once a file's in the metadata should be distributed as widely as possible so when i do this it has some special way of pushing the um so to my original demo repository it synced the skid annex branch and then it synced mastered there so let's go back to the original repository oh i'm not trying i got it yeah so maybe you show it yeah go back to the yeah yeah so this is something that took me a while to understand but really i think the important thing is that when you run git annex sync then it sends the metadata everywhere okay so in all the all the different uh yeah but how does it know how does it know well so each repository has a few remotes whenever a repository runs sync it finds the ones it's connected to and then gets the metadata and then sends new metadata to me it's still sort of a little bit magic so um so does it push like the metadata to all the remotes is that what's happening this is what i understood yeah yeah is it correct yeah so here we see there's get on this is the graph of the git repository so we see there is the master branch here and then we see there's the git annex branch so the get annex branch stores the metadata and it's been now sent to origen okay yeah so i think that maybe we should go on um there are also a few questions on here you can see that okay so one question was maybe so in the current example if someone would clone the repository the person would get the metadata but could not use the files right yeah unless they have access to one of the remotes that has the files stored at them let's go back to ga demo uh okay uh let's list here we get annexed list we see so here this doesn't know of the other remote yet uh if we get annex sync actually i don't know if this will work okay so [Music] it doesn't yet um because it's not connecting yeah how you cannot know right yeah so we need to add the we add the other annex demo as a remote now yeah then yeah and now we see others let me see here so let's say we don't need some files on here anymore okay i just dropped file one it's no longer in this repository let's see let's let's drop the large file uh there's an error message so it could only verify the existence of one out of zero necessary copies so it won't let us drop it we can but it suggests we move it somewhere else and we can use force to override the check so this comes from git get annex num copies which is by default one so it won't let us drop data and lose it completely oh yes i got it yes but that's good i mean that's quite secure in that sense yeah so let's do what it suggests we get annex move large file somewhere else list so now the two files are on other and not here and i can get them back so yeah so this is the way of moving the data around using actual git links so already what we have here let's say you have you're a group working together and you already access data via ssh or something like that you can start using git annex to move the data around you can have the working copy on the cluster another file server for archive and then you can tell the archive i want one copy of everything there's a way you can define what each repository wants like you can say the archive wants a copy of everything so here i'm running git annex sync content and i think this will get all of the files here so by default the sync content means bring all of the files to every repository but i can override that but that's getting something to be quite advanced let's talk about it at the end instead of now let's go to something cooler which is the special remotes and this is where we get to the cloud storage and things like that any questions so far let's see so there is a question about compatibility layer you get lfs and get annexed the other question which maybe we should also do now so and again coming back to the actual files so the one that i'm already know so to upload the actual files we use the normal git add git commit in my understanding no right because we we never want to actually track these with normal gifts right yeah so how are they really stored how does that really work but maybe you will come to come to that because it's still not completely clear to me well but this is what your is it i thought you explained it at the very beginning with the dot git yeah so when i move a file from place to place it's shuffling stuff around inside of the objects directory here somehow magically behind the scenes and making the siblings work but is it really tracking changes in the real sense of changes if you change like a binary an image well i guess we can try that so get so i'm gonna get annex unlock file one and let's edit file one new line and then i do get status type changed file one so it turned from a link to this so i think if i get add file one it will complain it didn't complain it doesn't complain so you can change it from a guitar next to so i think if i do git commit it will make it annexed again because i had i had unlocked the file before so let's see git commit and look there to sim link again so if i do get log then i see yeah it is back to initial commit large file that so all of the syncing and stuff happens in the git nx branch and this is only tracking the changes in the files themselves if i do so we work as usual actually yeah pretty much if we look at the git annex branch we see well a bunch of things that says update this is sort of just transparently handling all of the metadata about what is where okay but let's go to a special remote so the weakness here is that anywhere you put data has to have a git repository and you have to have shell access to it like this isn't working for github so let's use what we call the directory special remote um let's see where did i do this here we did so let's make a new directory so this is file storage and why are we using file storage well it doesn't really make sense to put another copy of the files on the same computer but this will show what the key value store is like and then you can imagine that this is being pushed to s3 or well any other kind of cloud service google drive whatever okay oh okay because i think this will clarify it for me because that's that's what was next yeah i don't know thanks so there's init remote so this is not named very well init remote means init a special remote which is not actually a git repository and this will be called directory storage type equals directory so let's say it's shared encryption so that means that it will create an encryption key store it inside of this git repository and then push the metadata and then whenever when the data is put in the remote it will be encrypted so encryption equals shared um file storage okay so here this is a real remote what do you do instead of the folder so we'll demonstrate connecting to s3 next if there's time you would basically say let's see the options you use are host equals a3s dot fee bucket equals whatever protocol so there's basically using these options you can pass whatever you need to it in order to set it up okay so get annexed list so now we see the directory storage so should we move the large file to there yes large file to the directory storage so this took a little bit longer because it's encrypting it oh [Music] but the encryption is because you have chosen to encrypt right yeah so if we look at the file storage we see there's a temp and this thing here so it's also encrypting the file names of everything uh in seven two three we see another thing so it's like a hashed or uh using file names to hash the storage and drill down some so that one directory doesn't get too many different file names and here we see so something says gpgh maxshaw1 let's look inside that directory and it's the same thing and let's look at the file with less and it says it's a binary file and we see something that ah this was the large file so of course it looks random let's copy the small file to it and instead of that we'll look at three two four [Music] look at this file so we see it's small but definitely encrypted compared to what we had here so where's the encryption key stored let's go look at the git annex metadata so show git annex so this is the contents of the git nx branch so we see these somehow store the metadata of the files and then there's remote.log if we look at remote.log we see this link here which is name directory storage type equals directory timestamp of when it was updated and this is probably the cipher key yeah so this may not be what everyone wants but this is basically being able to clone the git repository makes it able to get the um get the data from the other remote so for some use cases this is useful so it's easy to control access to a small git repository it's hard to control access to large data which may need to be stored on some other specialized service and here this turns the hard problem into the easy problem there's other ways of doing the encryption like um using gpu g keys so instead it encrypts the cipher with the key with your personal key and then stores it in the uh remote log so you need both the git repository and your own personal key in order to access it okay so how does this store stuff in cloud and things like that well if we look here it's basically like this so this becomes a key value storage and then it puts whatever it needs into s3 with these kinds of names or in google drive or in a directory in the case here there's a way you can make special remotes by making an external script which will which it's a protocol so you can write any kind of um special remote you would like so when you're first trying to use the previous csc storage system i wrote something called get annex remote irons um which would store it in an eyebrodz remote which in the end we never used because it got replaced with alice but that's the basic idea here okay so what's the question that came in also i don't know if you ever tried git annex web so that then you can have a web interface right remotes and synchronization we could have a look at it we have time yeah and my question that i had was that so let's imagine that your research group is managing data with gitanex and the data is stored on s3 wherever and now i come in a new group member so i would have to get access to the git repository i cloned the git repository and i also need to get access to the to the storage and then what do i do so then you would well you would clone the git repository and then to your personal workspace and then you would presumably you would know what file you want to work on so your group would say okay you need these files so you would do git annex git and then it would know from the metadata that it should be stored in a special remote with this host name and this location so you can also embed the s3 token inside of the repository metadata so it would have everything you need in order to access it all but um perhaps you want someone to have to get a separate key to do that it depends on the data i guess yeah okay do you want to try the s3 thing quickly or are we okay let's do it so the hardest part so okay now i'm going to try to use this in order to store data into s3 in the csc service called alas csc is the finish national um e-infrastructure and computing center so they have an s3 storage which is their recommended place to use data the problem is in order to use this your idea is okay i'm copying this data here i'm getting this data to this directory but how do you track versions of things how do you know what's where how do you not forget something so this is where i think git annex can be useful and the hardest part of setting this up was to figure out just how alice worked they provided these scripts and it says run this and then you can access it but it doesn't say here's the s3 host name and here's how you get the token so i did this about two hours ago let's hope it still works oh so it made this file actually i had to modify it a little bit and then uh let's put the tokens in there get annex init remote type equals s3 i'll do encryption equals shared because there's really no reason not to host equals a3s dot fee bucket equals ga demo 2 because i made gi demo 1 earlier today and then i realized i had to do these two things here and let's hope this works why is it pulled so it get annex set it would be able to find the right port if i specified the protocol but somehow it didn't so i don't know and let's not forget to tell it the name a loss oh yeah okay otherwise it's quite simple yeah i mean well it looks simple once you have the s3 credentials but getting those s3 connections was quite a challenge okay but on a normal amazon it will it's traced for a while to get this yeah um and also afterwards it would be nice if you can share this how to how to do that so that's the question on imd right how to set this up with unlocks because many will be interested in that yeah i will do that okay so now i can get annex well i wonder if this works no i guess uh copy okay next copy the current directory to allah so it's encrypting [Music] okay and now it's uploading i guess it will take a little bit of time and now there it is so you may think it's a little bit of a problem that the file name in alas won't match the file name that you're using on disk so just looking at alice doesn't give you enough to access things get annex has in the last few years developed something called import 3 and export 3 which means that you can export everything with the real file names that you see here so basically you're exporting it um how would you call it like you're exporting it with real file names the problem is that's an option you said before you do this copy right yeah now it's too late to do that this would be a different special remote so of course i could add another alla special remote with a different bucket that uses export 3 to save the file names there's an option within this to tell it to make the s3 bucket public and then i can tell in git annex that this is the public url of the bucket so anyone by cloning the repository will be able to do the git annex kit without getting their own s3 token which might be useful for some use cases i was thinking exactly yeah to use it yeah but that's really nice yeah you could even have several remotes one for your internal group work which is just the like everything and one for the data you want to make public and then you put it there okay so here i had opened the csc web interface uh with different this is a test project we have ga demo2 so we see there's some file called annex uuid so it'll be able to know what the bucket is and it has the gpg encrypted stuff here it can also do chunking of the files if the file is too large it will break it into 100 or 50 or one or however many megabyte chunks you want so let's see um yeah and that's basically what i wanted to talk about oh it's done let's refresh so here we see there's two files 100 megabytes and 89 megabytes so it's there with a file name that can't be understood by the remote and content which can't be understood by the remote let's do a metadata test [Music] let's see how this works i've never actually used this metadata path set [Music] set so now i've set the owner of something to that so now whenever this is being cloned around you can say record who's managing each file or things like that when it was generated the experimental parameters all that kind of stuff nice for the problem yeah and i think i've even seen some ways that whenever you add a file it will um you can automatically generate some metadata based on a script when you add things but this is like beyond what i've done oh yeah yes okay so you could keep track of uh i mean some additional metabolism yeah nice that's really nice do we have anything like that in norway that we can connect to and hook up what just happened i'm using the real amazon yeah with my own money okay i just ran git annex map which is something i saw and it made this little link that shows what knows what which is interesting oh nice yes so we have a couple of more questions on one yeah so the question was about encryption so if the repository is public then having the tokens keys in the metadata would be problematic and i think this is something that we have also been talking about on zurich just before the stream yeah and maybe related to it or so are encryption is this encryption requirement or is this an optional thing okay encryption is optional um it can be none shared or public key i think there's something called hybrid but i don't remember what that does um okay what other questions the other question was does it use s3 cmd in the background or swift i don't know so i think s3 is built in so it might use a haskell library for s3 stuff um let me show you something so here's a thing called get annex remote rclone if i look at this we see it's a large program and well actually i don't know what any of this means but i can somehow okay i have to look it up now so let's go to the get annex web page special remotes so here's all the different options it has well let's see and while we look also clarifying question so in this case only people who have access to allah's bucket would be able to access the data right correct yeah i mean there's one kind of these multiple different things going on there's where's the encryptions key stored who has access to the bucket like if you want other people to access the data and you have a shared encryption key then that's not really a good idea but presumably you would also [Music] well only copy the data to a public remote that you wanted someone to be able to get so like there's basically so many different options for how you can design these things okay but we scroll down and there's these um other things so where's rclone our clone special remote so it accesses all of these different cloud services using the script that i just showed you um [Music] and let's see how we would set it up okay i see we on cloud is supported yeah so can we have one cloud in norway well there you go uh you can also tell how trusted remotes are so you can say alas is a trusted remote which means that git trusts that the data is actually there before dropping it or you can say something is untrusted which means that you expect the remote might drop the data at every time at any time so it can count towards the minimum number of copies you want to hold okay so here's how we would set up a special remote git annex and it remote the name type equals external external type equals rclone which means it will find the skid annex remote r clone and then whatever other parameters you may want for it okay so before we started i said there was about eighty percent chance chance that you thought this would be what would i say um [Music] yeah i give an 80 chance that you think this is one of the coolest things ever so what do you think i want to try it honestly on a real project yeah so one of my recommendations is ask someone for help maybe me maybe someone else because there's a lot of documentation on it and while it does seem to be rather complete it needs well it took me a long time to get to where i was now maybe now though with this introduction it will be a lot easier for you to do things yeah also maybe like a blog post write up or something just will be i think super useful yeah i think if i try it on a real project i can write a blog post yeah yeah i don't think you can right now or if someone else will use it on the real project i think it could be very cool yeah i mean so i think this really solves the problem actually it was at the research data alliance meeting in helsinki over the no in the spring no last year so i was at this presentation that talked about the difficulty of getting people to use cloud services or these kind of e-infrastructure services so they said that once you get to the point of here's an object storage this is the modern way of doing things use this for your research well you basically have people that need to write a data like what's the word data management data replication layer in order to really use that efficiently and only big projects do that so git annex sort of is that data replication layer so it replicates the data it remembers what's where it pushes pools it prevents you from deleting things you don't want and that's what we really need who is behind this is it like people are working on the free time or is somebody really funding it so it's made by a debian developer called joey hess um it's maybe 10-ish years old something like that and it's mostly him for the most part there have been some crown crowdfunding sources to get um actually it was crowdfunding in order to fund development but also it's been a part of some major grants like this data lab project that enrico was linking to so that was a collaboration between research groups and joey to develop a service using git annex that would be able to replicate data all over the world like this so it has some support but really i think it could use more yeah it looks quite i mean simple once you have all the procedure right yeah to put in place uh i really would like to i will try yeah to use it on a real project to see if this is as simple as you should yeah two more questions if we have time what is uh what is the record the largest file or largest amount of us we have ever tracked with gitanex and the other question is so if if somebody already has a lot of data in alas bucket or somewhere on amazon can you link it with the repository how would you start to start using git onix on something existing that's a good question so for the first one when i was going to use this for some data i had someone asked well does git track that many files well and according to the git annex docs the limit with number of files is basically the limits of when git starts slowing down um for how large a file can be i don't really know of what the limit would be actually let's see i think i've even there's a scalability there is a limit because it doesn't really track i mean if you put things on the yeah bucket it's usually quite large and here it doesn't work much the for the size probably size limited by the storage of where that is yeah yeah but also so that's what i will try i mean we have right not very large in that case but also because git annex can do chunking in the special remotes so if the file limit size limit is one gigabyte or something you can tell it to chunk all the files to 500 megabytes and then incrementally up and down load them in that size chunks so you can actually go from large files locally to small files stored somewhere else so you would even improve performance in that case yeah so it says arbitrarily large files can be managed memory usage should be constant except for memory links many files can be managed and i did a test where i was making git repositories with the order of millions of files in it and get annexing them and i just couldn't make it stop working really well like it worked it seemed to be mainly limited by disk access speed so okay um we have time to comment on that question so how do you start how would you start if you already have data somewhere okay do you mind if i try to do something live no okay so i want to look at this import tree thing so we get to the problem of not knowing where to click and there's no contents here let's see internals i can already announce in the meantime that probably we will not have cluster etiquette right yeah probably not which i'm really sorry but then we will do it next time it's something i really want to talk about yes does get annex import doing so here importing from a special remote importing from a special remote first downloads or hashes all the new content from it and then constructs a git commit that reflects the files that have changed since the last time it was looked at merging that commit into your repository will update to reflect changes in the special remote so to me this implies that it is possible but we're sort of out of time so it depends on how much you want to see me try to do things but you can try it and put it in the blog post yeah i think this writer will be really really useful i really like a walkthrough of all all the all the steps and thanks also so much for lots of questions today when i can do it was really great three i wonder if this will work it's directory oh we have to say in encryption equals none so the encryption options are non-shared hybrid public key or shared public key okay there it worked oh yeah okay nice so how does git annex import what work okay back to drawing more but maybe not everything can work with import three i guess it depends on if the special remote can be listed because even being able to list the files in a special remote isn't one of the requirements of using this to import from import master from my remote let's see [Music] oh what's the remote name oops okay i called that wrong uh three let's call it three yeah there that's funny okay oh yeah hmm okay that failed from what did it say here embedded option from well it looks like i did exactly what it says there maybe something is out of date ah non-exhaustive list so it would work with the s3 report [Music] maybe we can read the help text what git annex so maybe i can mention a little bit unrelated question that came up on hackmd just while we look for the solution here and that is i think it's a great question and that is do we have recommendations for good scientific software podcasts or python podcast oh yeah rsc stories how could i forget yes oh yes that's very good yeah that's great yeah we mentioned it a couple of episodes ago yeah that's really nice one i i don't know the other one talk python to me i need to check that out yeah i need to check it out looks nice i wonder if it could be that oh nice oh this is that's all right not i had just updated this i updated this right before um the show oh yes it's doing something list three so it's only available in your version or you had a pretty old one before hmm well should i give up or keep going i mean otherwise we can try on flying yeah or maybe we can summarize a bit and thanks again for lots of questions and i can be that was really pleasure yeah sorry we didn't manage to talk about cluster etiquette but then like you know tv series we leave in a cliffhanger next time next year think maybe we can do a whole session on really clustering because there is a lot to talk about yeah yeah and okay this is really cool and we are waiting for the blog post yeah thanks everybody yeah thanks for um guiding me through this okay so do we stop now or any more questions i guess we basically showing us this it's very cool actually okay so i'll stop the recording