Distributed Databases on the Jamstack

Captions provided by White Coat Captioning (https://whitecoatcaptioning.com/). Communication Access Realtime Translation (CART) is provided in order to facilitate communication accessibility and may not be a totally verbatim record of the proceedings

JASON: Hello, everyone. And welcome to another episode of Learn With Jason. Today on the show, we have Natalia. How are you? Thanks so much for joining us.

NATALIA: I'm doing great. I'm just, you know, not ready for the winter that's coming. But doing really good. How are you, Jason?

JASON: I'm doing great. I am having a little bit of a frantic morning. I just realized as we were getting on that I, like, hadn't installed some of the stuff. So I was scrambling to get that to work, which is why we're starting a couple minutes late today. But I am really, really thrilled about what we're going to do today. I think it's going to be amazing. But before we talk about what we're going to do, let's talk a little bit about you. For folks who aren't familiar with your work, do you want to give us a bit of a background on yourself?

NATALIA: Sure. So basically, my background is in design, but that was quite a while ago. I think it was about 15 years ago that I moved or transitioned from being a graphic designer to being a web designer. And then a developer. Then I've been doing mostly front end for the greatest part of my career. And then also other technical -- I don't like the term full stack, but going from one side to the other of the stack. For, I think, the past four years, I've been in the role of solutions architect. Now I'm doing this MongoDB. So this is why I'm here. I want to talk to everyone and tell the folks how they can better work with MongoDB and get them all started.

JASON: Yeah, and so this is -- I think what's exciting about this is we're talking about MongoDB, but we're talking about MongoDB in a context that I think is somewhat new, right, which is how do you use MongoDB in a serverless environment.

NATALIA: That's right, that's right. I think this is one of the misconceptions. It's not a misconception, but it's a lack of understanding or perhaps of knowledge because MongoDB has grown so much, and it has evolved so much and changed and adapted to the development needs. Right now it's not only a database. It's a data platform. So in the cloud, we are offering a really, really good development environment for people to create and to, yeah, do amazing things.

JASON: I'm really excited about this because I'm gong to be honest with you, I have not used MongoDB in a very long time. The last time I used Mongo, I think it was just a database. It was kind of document storage. That was, oh, geez, it had to have been five-plus years ago that I used it. So I am really, really excited to see the progress and how far it's come. I'm also excited to see -- you know, one of the things I think is exciting is the way that databases are evolving right now is in such a way that it feels a lot less like we, as front-end developers, need to learn how to become DBAs or full-stack developers or back-end experts to create and use databases. It feels much more approachable in today's world to build really complex apps.

NATALIA: Absolutely. I think as a person myself who's coming from design, just imagine something completely different to being a developer. I think one of the things I was scared the most about were databases. You think about this as a, I don't know, point of failure, as a security flaw, perhaps, as something if you do wrong, you are going to probably degrade performance in an impactful way. I think you're absolutely right. What the cloud and these serverless environments allow us to do is work with databases in a way that doesn't feel really magical. You know what's going on, but at the same time, it's not an environment. It's not something that you -- yeah.

JASON: You just listed off all the things that you could worry go wrong, and first of all, how dare you go through my list of secret fears. (Laughter) But we -- I mean, you're absolutely right. I think that as a front-end developer, I feel really confident shipping to the web. If I ship a broken piece of HTML or I have a small JavaScript error, I'm fine. We can recover from that. It's easy. If I ship a bad database query, I'm like, wait, but I could delete the whole database, right.

NATALIA: Exactly.

JASON: And I've done that before. So -- (laughter). So I'm very excited.

NATALIA: Yeah, back in the day it was super difficult to recover a database, which is not the case right now.

JASON: Exactly. Yeah, and part of it is I think the tools mature and get better and get less intimidating. Even now for somebody who wanted to set up and manage Kubernetes, it's not as daunting as it may have been, you know, five years ago to work with containers and massive scale and stuff. It's much more approachable. But it is a whole area of expertise. I think databases are kind of the same thing. It's this whole area of expertise. A lot of times, I want a database not because I'm building Twitter and because I need to handle billions of records and global scale and all of this stuff. I have an idea for an app, and for my app, I want people to be able to submit data and share it to other people, and I want that to persist between page loads. That's what I want from my app. So I don't need to be a database architect. I don't need to know exactly what the best thing is. I just want to be able to click a button and say, can I put some data here? And the database high fives me and says absolutely. What I love is that it feels like tools are starting to -- that's the default. You start there, and then you can customize as your needs become more intense.

NATALIA: Exactly. That's exactly the point. Particularly with document databases. I think it's worth it if we highlight the difference between a relational database and a document database. So a relational database is mostly tabular. It consists of tables that are related by what is known as a foreign key. So you can make relationships between data. And a document database basically stores all data that's related in a single document. Of course -- and this is one thing that most people don't know -- you can also create relationships in a document database.

JASON: Right.

NATALIA: It wasn't designed for this use, but obviously, there are use cases that require that you create these kind of relationships. Then these modern databases or general-purpose databases like MongoDB allow you to do that as well. But the whole concept is it's basically JSON. And that's what we as front-end developers are so used to working with. It's what we get from a response to a request for an API. We're very familiar with it and know how to parse it and iterate it. So for us, it's very natural to work with this kind of data. As you said, it requires the expertise to create the right queries, to use the NQL, which is the Mongo language, and also the aggregation framework that we're going to take a look at today.

JASON: Great. Okay. Well, I know we've got a lot to cover. So I want to make sure we get into the code and don't run out of time. So I'm going to switch over. While I'm doing that, I'm going to do a quick shout out to Salma, to Ben, Twitch coder 7. Thank you for coming out today. Good to see you in the chat. I'm thrilled to have you here. Let's talk a little bit about our live captioning. This show, like every show, is live captioned. We've got Rachel here from White Coat Captioning taking notes and writing everything down as we say it, which I very much appreciate. And that is made possible through the support of our sponsors, Netlify, Fauna, and Auth0 are all kicking in to make this show more accessible to more people. You can find those right on the home page. We're talking today to Natalia, who is anfibia? Is that right?

NATALIA: Yeah, that's right. You got it.

JASON: Sorry. These are the questions I should ask before we're live, right. (Laughter)

NATALIA: It's fine. (Laughter)

JASON: So make sure you go and give her a follow. And now, we are going to be talking about MongoDB.

NATALIA: Yes, and maybe you want to go to cloud.MongoDB.com.

JASON: Aha, all right. Here we go. We're at MongoDB on the cloud.

NATALIA: And we want to try it free. So you have that button over there.

JASON: There we go.

NATALIA: Now you are going to have to create an account. You have that form on your right side.

JASON: Okay. Sign up. Here we go. Yeah, we'll link my accounts. Oh, I apparently have a MongoDB account. Give me one second to pull this off screen so I can figure out what I've done here.

NATALIA: Sure, let's keep secure.

JASON: Make sure I don't have, like, some kind of production data in here I need to worry about.

NATALIA: Sure. No, and you may have to do this a couple of times if we get keys. Let's see how far we go with this.

JASON: Oh, you know what this is? I was so prepared. You told me before that -- I think this is it, right? That I should set up this account.

NATALIA: Yeah, perfect. That's great.

JASON: For the first time ever, I'm prepared for something. (Laughter)

NATALIA: Well, that's wonderful because we are saving time. What I just want everybody to know, this is a free cluster. This is a sandbox. You have loaded the same data. That's fine. We can show folks how it's done. So they can play with data as well. But this is free, and this is free forever. This is one of the amazing things about MongoDB in the cloud. You don't have to, like, Netlify, you don't have to pay for anything if you are going to be playing around and learning how to do things. It's not necessary to pay or set up a payment method.

JASON: I was just going to say, this is such a great -- like, I love that we seem to have kind of converged on this as an industry. Learning is not something you should have to pay to do. You should be able to get in and try things and figure out how they work and then when it reaches the point that you are using it professionally, that's when you flip on the credit card and the company services and the things that you pay for. You get value out of the tool, and the tool takes a percentage of that, right. If you're making no money, if you're learning, if you're just putting up a side project, then it doesn't make sense to pay for that in most cases.

NATALIA: Absolutely. And also, when you have the chance to learn a technology, you become an advocate for that technology. How can you want to use something you don't know how to use? So I totally agree with you. So if you go back to Atlas on the top, you see that we have this overview of our database deployments. In this case, you only have one cluster. What you did, and this is why you have data, is if you go to these three dots, it says load sample data. This is why you have some collections and some documents in your database. What we're doing to do is we are going to use the Mongo shell you already installed as well to import sample data that I created specifically for this case. The interesting thing is that there is a tool when you are working with a database and you want to use your own data, MongoDB has a tool that allows you to do this. We just create a schema and use this tool called Mongo Generate that we're not going to use right now because I already generated the JSON files. That allows you to create data as you need it. Because one thing, it's giving you some sample data you can play with. Another thing is you are building a front end and kind of need some representation of what you're going to render.

JASON: Sure.

NATALIA: So if you go to the repo that I sent you the link for --

JASON: Yes, that is in here. I just have to find the link. Give me just a moment.

NATALIA: Sure.

JASON: It's the tattoo app, right?

NATALIA: Yeah, exactly.

JASON: Okay.

NATALIA: I'm going to make this one public, but after I check it. I always like to check that there are no things that shouldn't be there. So if you go to -- let me see. There should be a folder. Can you scroll a little bit? Database exports. Yeah, that's it. So like we mentioned before, MongoDB, when you're working with MongoDB, you're essentially working with JSON. But how the database stores data is in a format called BSON. It stores it in this way because it has a lot more types than JSON.

JASON: Got it.

NATALIA: So because we can work with JSON or BSON, there are different ways you can import data into your database. If it's JSON, it's going to be Mongo import. If you want to export data out of your database, it's going to be Mongo export. So in the commands.md, I already wrote for you the commands we're going to use to import this data into our database. Let's take a look at what this command says. So it's Mongo import and passes a URL that we are going to get from the cluster. This is our command string or connection string. I apologize. If you ever hear people say, oh, I need my connection string for MongoDB, it's what it is. Then it also establishes what is the database name. In this case, tattoo network. If it doesn't find it, it's going to create it. Also, the collection, if it doesn't find it, it's going to create it. It will tell you it's JSON because it could be CSV, for example, or any other comma-separated format. Then the file. So just copy the second one. And if you go to -- if you clone this folder or download it --

JASON: Oh, yeah. Let's get it now. It is -- so we'll start here.

NATALIA: Oh, that's nice.

JASON: Thank you, Wally, for the sub. Let's look. Atlas tattoo app. Here we go. Then should I run an npm install?

NATALIA: Not yet. Well, this is the Angular application. I think we can go there later. Let's set up the database first. If you go into this folder in your terminal, if you go to the exports --

JASON: Ah, it's right here. I see.

NATALIA: Exactly. So you have to run this command exactly where this file is at.

JASON: Got it. So let's get into the tattoo. And we can see here's our file. So now I can go take this command --

NATALIA: But don't execute it.

JASON: It's not going to work yet because we need to build an actual --

NATALIA: Exactly. So what we're going to do is go back to the cluster.

JASON: Back to the cluster. Here.

NATALIA: We are going to click on connect.

JASON: Okay. This seems right.

NATALIA: Yeah, so this gives you all the possibilities. You can connect with the shell. You can connect with Compass. And you can connect -- well, what is the other option?

JASON: Oh, yeah. How do I go back? Here. Connect, shell, and compass.

NATALIA: Or from the application, right. So if you have a driver between your application and MongoDB, which is necessary for both processes to talk to each other, then you can also get this. Now, because you already had an account, what I'm not sure you do have is network and database access. So before you even copy this, we are going to make sure you have a user, and we're going to make sure you have restricted the access to your IP, which is a best practice.

JASON: Okay.

NATALIA: So just close this pop-up. And go to database access. That's on the left. That's right. Under security. You have a user. I don't know if you remember the password. If you don't, maybe you want to take this out of screen.

JASON: Let me try this one again. (Laughter) all right. Let me set up another one. Okay. So we'll use a password. The username, all right. Let's do this. Then for the password, I will --

NATALIA: Don't say it.

JASON: Got it. Then we can go -- all right. So it didn't show it. I can put these in. Password authentication. Built-in role.

NATALIA: Yeah, so MongoDB's databases' authorization methods are role-based. It has roles already built, but you can create custom ones. In this case, we're just going to use this. Read and write to any database. So you're going to be the database administrator.

JASON: Right. So we could add some extra stuff.

NATALIA: Yeah, we could create a custom role. We could restrict the access to specific clusters or even data. We could create a temporary user if we have a friend and we want them to work with us. Then later these credentials are eliminated from the system. But that's enough. You can add the user. And another thing we want to do is go to network access. And you are allowing right now access from anywhere. We don't want to do this. So just delete this. Or edit.

JASON: Okay.

NATALIA: And let's say add current IP address and confirm. This is going to make sure nobody else but you or somebody sharing your same IP address can access the database.

JASON: Okay. I'll make sure my partner doesn't know she can hack my database right now.

NATALIA: Yeah, that's always a good thing. So yes, it should be quick.

JASON: Okay. So then when we go back to here and we hit connect --

NATALIA: Exactly.

JASON: Right. Then I can put in this shell.

NATALIA: That's already your connection string. Yeah, you can copy it.

JASON: So I drop this in here. But then I need to add my username at the end.

NATALIA: Exactly. No, no, no, no. You don't need to do this. No, no, no, no. It's going to ask you.

JASON: Oh, perfect. Okay. So I can just run this?

NATALIA: Or, let's do this. Let's connect to the cluster. No, let's just go ahead with this. It will ask you.

JASON: Okay. So I'm shipping this command?

NATALIA: Yeah.

JASON: Okay. No! Why? What have I done?

NATALIA: Can you run Mongo? To see what happens.

JASON: Well, I have Mongo shell. Do I need something else?

NATALIA: Now you're connected. Connect refused. It doesn't even like your local host.

JASON: Maybe I had something running. I might have had something running locally. Do I need to log out or something?

NATALIA: Just open a new -- just open a new tab in your terminal. Okay. Let's try to connect to the cluster without doing the Mongo import. So go and copy the connection string, just as it is right now. No, no, no. From the connect.

JASON: This is the one from the --

NATALIA: I think it has a little bit more to it..

JASON: And do I want to -- all right. Let's just fail, try again, and I'm going to set my username here, just like that?

NATALIA: Exactly.

JASON: Okay.

NATALIA: And now it's asking you for the password.

JASON: Connecting to. Okay. So now it says I'm connected.

NATALIA: Yeah, exactly.

JASON: Wait, wait. Oh, no. This is right. Now I just exit?

NATALIA: No, no, no. Don't exit.

JASON: Oh, like run. Got it. I think I was -- yep, I was in the wrong place here. So let me get my actual connection string, which is this one. Okay. I think I've done something wrong. This looks like a MongoDB query shell, right?

NATALIA: Yeah, just exit this again. Because you shouldn't have to be connected because you're passing already the connection string. So this is what is going to prompt you to pass a username and password. Let's see.

JASON: So something is up with this Mongo import command. I only installed Mongo shell.

NATALIA: Can you try instead of -- can you tell me again the version? These things happen. Real life.

JASON: So this is 1.1.6.

NATALIA: Okay. Just let me see where I'm at. Yeah, because you downloaded -- okay. It may be because you're in a very, very old version of Mongo.

JASON: For Mongo import, I need Mongo tools, which I did not install.

NATALIA: Okay. Yes, that's right. I never do this. Yeah, you're going to need more stuff to make this work. And it's a shame we cannot do it.

JASON: Yeah, my brew command has fully -- like, something is wrong with my computer. It will not run brew right now, which is super fun. So I have to figure out what happened there. In the meantime, we're going to have to figure out how to install this.

NATALIA: Don't worry. We're going to -- yes, so you need to have -- so you have this server, but you also need tools. Import tool is part of the MongoDB database tools package. And you are going to have to download it as well, installing the database tools on OS.

JASON: Here's the tools. I'm on this MacOS. So I'll download, and then we'll go back here and figure out how to do all this stuff. So we're going to go to --

NATALIA: We wanted to be ready, but we couldn't make it.

JASON: This is my fault. I should have tested that brew was going to work. Who breaks brew? Apparently I do. (Laughter) so I'm going to unzip this. Then we're going to move into the downloads folder. What was it called? MongoDB database tools. Okay. Then what else do I need to do here? I need to copy them -- so, I need to copy the whole bin folder, right?

NATALIA: Exactly.

JASON: So let's copy. Then we'll source my Z shell. And theoretically speaking, that gives us -- where's my -- oh, here it is. Okay. This is better.

NATALIA: Authentication failed. So that's probably --

JASON: So is this like I need my -- like that?

NATALIA: Yeah, enter password. Exactly.

JASON: Ta-da! So we got through me being woefully unprepared for doing that.

NATALIA: No, it's okay.

JASON: Now, theoretically speaking, we should have in our collections tattoo network.

NATALIA: Exactly. Now you have artists. And can you click on it? You have 1,000 documents. The document size. Everything is there.

JASON: Oh, this is great. Okay. So now we can do the same thing --

NATALIA: With the other one, yeah. You probably don't need to connect because you're already connected.

JASON: Okay. Let's see if it'll work. I'm going to go to opinions. Now it looks like I'm going to have to do it again.

NATALIA: Oh, okay, okay.

JASON: Something I've done has gotten this weird. Uh-oh. Oh, wait, did I do the -- crap. I tried to import artists twice.

NATALIA: Okay. Ah, right. Yeah, this is data that I have generated. The ID is already generated. This object ID. This ID obviously cannot be repeated. It's something that is unique. If you try to override it, it's going to tell you, hey, maybe you don't want to do this. It's a duplicate key. So did you get the opinions?

JASON: I thought I did. But it's not --

NATALIA: What does it say?

JASON: Browse collections. Tattoo network. Here is the tattoo network.

NATALIA: Can you try again, the opinions one?

JASON: Yeah, let's try it again. I'm just going to double check before I -- so here's our connection string. We're going too tattoo network. We've got a collection called --

NATALIA: Ah, collection. Now we have a super huge artist collection.

JASON: I should probably go delete that, huh?

NATALIA: Yes, yes.

JASON: Okay.

NATALIA: So now you have -- yeah, you have 6,009 documents in your artist collection. You can delete it here, or you can delete it from the command line.

JASON: Let's delete it from here to make sure -- here it is. There's a trash button.

NATALIA: Yeah, exactly.

JASON: Then I will go back to here. And we're going to -- here's cluster, artists, artists. And that should theoretically speaking, now we have our artists and our opinions. And there they are. Okay. I imported them all to the right place. 1,000 documents here. 5,009 here.

NATALIA: So when people talk to us, people that work at MongoDB, about how they're on monolithic applications and they want to decouple and how can MongoDB help them, usually the idea is that if you are splitting into micro front ends, each one is an application. It has a team and they own the architecture and own the tech stack. In some cases, they also have a database that maps to this team and application. Makes sense. But not all the time it makes sense to have one database per application. That would suppose that you have to duplicate data over and over and over. Although duplicating data is not necessarily a bad thing, you have other ways in the cloud with MongoDB to separate concerns and separate data in a logical way that doesn't involve having multiple clusters or having data separated and then having to, you know, rake your head about how you're going to aggregate all this. And we are seeing this, right. We have a cluster, and we have several logical databases separated, but we can work with them and we can aggregate the data coming from the different databases and collections and then create different strings or users. Any service can log into the database with a user that maps to or is governed by a role. This is how we can have a centralized, in a sense, more manageable set of data that we can distribute but in a smarter way than having, I don't know, thousands of clusters all over the place that we cannot control.

JASON: There's a question that just came in from Tyler. Is there a way that you can guard against like putting the wrong data into the wrong place? If I wanted to, could I set up some kind of a guard that would check the artist table to make sure that if I was putting something that was an opinion into the artist table, it would be like, hey, that's not allowed. Is there a schema check or something you can do?

NATALIA: Absolutely. There is a schema validation. We don't have one because we just created this collection. And we are not enforcing a schema in this case. And this is why it allowed you to upload all these opinions to the artist collection. But if we created a schema and enforced it with a JSON validation, it would have told you no, this is not allowed for this collection.

JASON: Got it. And that's great because -- and here's what I think is cool about this. The power of a document store versus a relational database is that when you're early and you're not 100% sure what your app is going to look like, having to redesign your whole database every time you make a change is really challenging. And being able to just have a completely unstructured document store where you're like, well, what if the data looked like this? That works and you can go full chaos mode. Then when you get it working, then you add the schema validation. Then you add the extra guards and checks. Which I think is a pretty -- you know, it's a pretty good experimentation model. It's a good iteration model that lets you start with, you know, you're in this completely unstructured space, no idea what's going on. Make things up, try things out, and as you hit a good stride and figure out what you want to do, then you start adding rules. Then you start adding structure. I think that's a distinct advantage of document stores. They support that growth.

NATALIA: Absolutely. It allows you to go from zero to a prototype that you maybe want to show to stakeholders. Look what I'm doing. How do you feel about this? Then iterate and make changes without having to work on ORMs, which are a pain, and all these more strict kind of constraints.

JASON: Absolutely.

NATALIA: So what we want to do right now is we are working on a tattoo network, right, in our heads. So we have artists that are part of this network. And then we have a collection that are opinions that people left for these tattoo artists. What we have to think is that imagine a context where we have an application, and then maybe the artist profile is owned by a team. So this page or this whatever feature is owned by a team. The opinions may be owned by a different team that also works with another team that is maybe distributing these comments and ratings to other parts of the platform. What we are going to build also here is a search application. So I'm a customer. I'm registered to this platform. I want to search for an artist, and we know -- I don't know about you, Jason, but I worked on several search application, and they are really painful. So MongoDB has come up with this idea to help developers create applications or search applications in an easier way. What we do is we are going to, in this case -- well, MongoDB Atlas search is built on top of an open-source library owned by Apache. It's used by Solar and Elastic search as well. So it's very well known in the industry. And it's very powerful. It has some differences with MongoDB, how it queries data that we're going to also take a look at. So what we want to do next -- and like I mentioned before, I went and created a guideline on how I want to proceed with this. So what we want to do is we want to show how we can create an application on MongoDB that is going to allow us to deploy serverless a GraphQL server to query our data in MongoDB.

JASON: Okay.

NATALIA: So we're going to go to this second tab that is called Realm. Realm is actually a -- just click on build your own app. It's the first part. Then you can call it something else if you want. This is the data source. This is your cluster. This you cannot change.

JASON: Oh, got it, got it.

NATALIA: But you can change the name of the application if you want. For example, tattoo app or whatever you want. Okay. We save it. The deployment model we can leave like this. And create your Realm application. As you can see, you have all these possibilities. What I wanted to say is that Realm was originally a mobile application. It was acquired by MongoDB to help with mobile development and also conflict resolution. I think in my opinion, the most amazing thing about Realm is the synchronization between devices. But we're not going to use that. We're going to use the GraphQL server, the web hooks, and the functions in this case. So let me see what it says. Just close this. I am not familiar with this, changes to third-party services.

JASON: Doesn't matter to me because I haven't used it before.

NATALIA: So it's great that we cannot see all your ID over there for the app ID. Don't expose it for now. And what we're going to do is we are going to go to rules. The first thing we need to do to create a GraphQL server and API is expose our data. So you see on the left side bar there is data access rules. We are going to add that collection. We are going to select our database. Collection name.

JASON: Okay.

NATALIA: And then we're going to select the template. Users can read all data but only write their own data.

JASON: That's slick. I love that just works. This is the thing I always have to configure, and I always have to look it up. So just having that preconfigured is huge.

NATALIA: Yeah, absolutely. Then let's call it -- it says each document. Let's enter tattoo artist.

JASON: Like that?

NATALIA: Yeah, yeah. Tattoo artist user. We're not going to be creating users, but this is --

JASON: Do we want to do it this way? What's the convention?

NATALIA: No, there is really no set convention for this.

JASON: No parents, no rules.

NATALIA: Yeah, but we are also not going to be creating authentications because it's too much.

JASON: Yeah, we are not going to have time for that today.

NATALIA: But it's required. So we are going to add the collection. And have you seen it says understanding save and deploy? Everything you do here now, until you save it and deploy it, it's not there.

JASON: And that makes sense, especially when you're working with data. If you auto saved every piece, you would get like, I have to make four changes and three of those changes would break my app because you're in an in-between state. That totally makes sense you queue up all the changes you want to make. Then you actually deploy that.

NATALIA: Exactly. You got it. So we have now this owner and nonowner. We could add another role.

JASON: New role, here.

NATALIA: Yeah, new role. So if you click there.

JASON: Got it.

NATALIA: No, let's not overcomplicated it. We just want to make sure we can read from the database in this case.

JASON: The non-owner will be able to read, right?

NATALIA: Yeah, exactly.

JASON: Let me trash this one.

NATALIA: Then make sure we have everything. Now what we want to do is we want to go to this schema. So we have rules. Then we have a schema. We want to say generate schema. Artist collection. So it's going to sample 500 documents in this case to make sure that it really creates a valid schema, that all your documents are more or less consistent. So you can go ahead with it.

JASON: Okay.

NATALIA: Generate schema. So it creates the schema. On top of this schema is basically GraphQL is going to create its own schemas to allow for all these queries and mutations. But because this artist collection is a very particular collection, we are going to come back here and edit it later. Right now what we want to do is create a relationship as well. For that, you're going to have to add the opinions collection first. So save this. You see the save button on top. Now you can add a new collection. MongoDB Atlas. There's a plus sign over there.

JASON: Here. Got it. Okay.

NATALIA: So now we can --

JASON: We're going to tattoos. Opinions.

NATALIA: Exactly. Same template. Same user that we did the other one.

JASON: Schema, generate.

NATALIA: Exactly.

JASON: All right. Okay. So I've saved these schemas.

NATALIA: As you can see, the types are all BSON types. As I mentioned earlier, we are using BSON because JSON has a lot less possibilities.

JASON: And is this format here -- is this, like, specific to MongoDB? Or is this something that's used elsewhere?

NATALIA: This is MongoDB. So if you realize it says, okay, the properties for this data, for the field ID, which is the identifier, is a BSON type object ID, which is basically --

JASON: So BSON is an open spec. Then this schema definition is unique to Mongo, built in BSON.

NATALIA: Exactly.

JASON: Got it. That make sense.

NATALIA: You got it. So now we're going to -- as you can see, we can validate. So you can a button there that would allow you to validate. Of course you have documents and created the schema on top of those documents. But if you were to add a lot more documents and you want to make sure they're valid, you could validate them here.

JASON: That's very cool.

NATALIA: Yeah. We're going to go back to the artist collection. And we are going to create a relationship.

JASON: Okay.

NATALIA: So for this, I'm going to ask you to go to the repository and copy. There is a relationship folder. So what we're telling here is there is a field that is called artist ID. If we go and browse our collection, we're going to see it's both in our artist document and in our opinion document. This is what creates the relationship between the two. For an artist that has an ID of 400, the opinion also has the artist ID of 400. This is what makes them relate to each other.

JASON: Got it, okay.

NATALIA: As you can see, it's called foreign key, which is how it's called in the relational world.

JASON: Got it. Okay. So I've now saved this.

NATALIA: And now if you go back to the artists and regenerate the schema, it's going to look, I think, a bit different.

JASON: Sorry, do I just hit generate schema again?

NATALIA: Yep.

JASON: Okay.

NATALIA: Artist ID. Okay. Can you then go to the opinions one? Maybe that's already generated, and this is something I got wrong myself. Generate the schema.

JASON: Do we need to include both -- like, do I need to create a relationship here as well?

NATALIA: No, no. The relationship is only necessary for one because this is the equivalent of a left outer joint. So it's from one table to another or from one collection to another in this case. It's okay because we have to make sure that our artist collection has yet another field. Perhaps where the difference is, is in the GraphQL schema and I'm getting them confused because it's so much information. I may have something mixed in my head. But let's save this. Let's deploy it. And yes, these things happen. I get confused all the time.

JASON: So we've got roles. We've got these. We've got properties.

NATALIA: Exactly. We have the schemas.

JASON: Here's our relationship.

NATALIA: Exactly.

JASON: Got it. Okay.

NATALIA: And you see it's in the form of a -- so you have the red parts and the green parts. And we can deploy it.

JASON: Okay. Deployment was successful.

NATALIA: Cool. Now if we go to GraphQL and go to schema --

JASON: We can start digging around in here. So let's look at artists. I'm just using control, space to kind of explore, which is one of my favorite things to do. So if I come in here, we can get, let's say, their name and maybe we can get --

NATALIA: The styles, for example.

JASON: Okay. And here we go. We've got a ton. That's great.

NATALIA: So we didn't have to do a lot. You have all the queries there. The first time I generated this, I didn't create the relationship right. So the schema in GraphQL was a bit different. There was no correlation between the two. All these fields that mention opinion were not there for the artist collection. You see over here. Yes, I really thought that it was also represented in the other schema, but it's not. It's only here in the GraphQL one.

JASON: Got it.

NATALIA: So you have the artists, and you have the opinion, right? You have a query. The mutations are below.

JASON: We have opinions. Then we can get the client name and let's get the rating. I will copy that out for a second so we can see the other one. So there's our opinions data. What was the other thing you wanted me to show?

NATALIA: No, you also have -- it's showing you the queries and the mutations as well, for people that are unfamiliar with GraphQL. You can, for example, delete one artist. I don't know what it's asking you.

JASON: So we've got all these. All that kind of stuff.

NATALIA: We are not going to use that anyway for the app right now. We don't have any mutations implemented. But we do have some custom resolvers. So data in GraphQL results to a type, right. What we want to do is create our own aggregations for this search application to work. What we are going to do right now is create some indexes. When you have data in a database, the smartest thing to do is to always create indexes. Indexes are like a set of information on the data. Like metadata on the data that helps you find it easier. Or in a more optimal way. The indexes in MongoDB are -- so you can have many nodes, or these kind of trees. But the indexes in Lucine, which is the search engine we're going to use underneath our search application, are inverted. So for that reason, the indexes that we're going to create are search indexes. It's a particular type of index. We have go to back to Atlas in order to create that.

JASON: Okay.

NATALIA: So we are going to go to the collections. Browse collections. The third one, yeah. And then if you go up, there is a tab called search indexes.

JASON: Here?

NATALIA: Yep. Just click there. And it says, well, we don't have one. The first one we're going to create is just a default index. A default dynamic index that's going to index every possible type that it finds in the collection. So we don't have to change anything. We just go with next. We call it default. That's fine.

JASON: Okay.

NATALIA: And we go next. And it gives us some information, what is the analyzer, the search analyzer, the dynamic. You're set. Okay.

JASON: Okay.

NATALIA: This takes a few moments. And the second index we want to create, also for the artist collection, it's an autocomplete index or an index that will help us run autocomplete. We're going to call it names in this case.

JASON: Okay.

NATALIA: So we're also going to go there next. And in this case, we are going to use the JSON. View JSON, over there.

JASON: I got it.

NATALIA: So this is what helps us create indexes in a JSON format. If you go to the --

JASON: Repo?

NATALIA: Yeah, to the repo, sorry. And find search indexes. There is one that is autocomplete type. As you can see, it's not dynamic because we don't want to index every field. We just want to index the field called name.

JASON: Wait, how do I --

NATALIA: Yes, false. It's not allowing you?

JASON: It won't let me edit. Do I need to refine?

NATALIA: I think that you were just looking at it, but you were not editing.

JASON: I think I might need to go back to --

NATALIA: Yes, you need to go all the way back. Back, back, back. Back.

JASON: This one.

NATALIA: Yes, JSON editor. That's right. So now we can edit it. Names is the name of the index. The field is name. Max grams and min grams are properties associated with autocomplete from three times you type until ten. It's going to be applying this autocomplete.

JASON: Got it. Okay. So our default index is created. Names is getting under way here.

NATALIA: Exactly. And because I'm conscious of time, we are like 30 minutes away from the end.

JASON: Yeah, we have like 25 minutes of good dev time left in us.

NATALIA: I think that we are going to have to do a follow-up, Jason.

JASON: (Laughter)

NATALIA: Because there's a lot to show. So we're going to quickly create also an index on the opinions collection because we will need it anyway. So you have the opinions index also.

JASON: Am I going to be able to do this? Because it looks like I'm --

NATALIA: You have two and the limit is three. So you still have one to go. So browse collection, go to the opinions collection this time. Ah, I think you are in the wrong collection. So you have to go back to collections. Next to search, you have the collections tab.

JASON: Here. Then we'll go to opinions.

NATALIA: Exactly. Now you go to search.

JASON: So it's telling me I've reached my limit here.

NATALIA: How is that possible if you only have two?

JASON: I'm not sure. Is there a way to view all the search indexes?

NATALIA: Yes, if you go back to all the collections, like browse collections.

JASON: Browse collections.

NATALIA: Yes. And click on search. Collection search should show you.

JASON: Ah, we have a movie search. That was for the sample data, I think. We can get rid of that one.

NATALIA: Yeah, okay. Confusing for everyone.

JASON: Now we go back to our collection. And we'll go to here and to opinions. And search indexes. Now we can create an index. All right.

NATALIA: All right. So JSON editor, and you can copy the opinion index.

JASON: Which is this one here.

NATALIA: Which is an index on the artist ID.

JASON: Got it. Okay. So let's go back in here. Do I need to rename this one?

NATALIA: These we're going to call opinions.

JASON: Okay.

NATALIA: And this is one thing that's important to mention. You need to remember those names. You probably will need to use them in your aggregations right now. So now we have the indexes. We can go back to GraphQL. And go to the tattoo app. That's right. And we're going to want to -- no, it's not yet going to work because we need to create the custom resolvers.

JASON: Ah, custom resolvers.

NATALIA: Yeah. So we're going to add one. We're going to create three of them. The first one is for the autocomplete. We're going to call it autocomplete resolver.

JASON: Okay.

NATALIA: Let me just make sure we use all the same names. It can sometimes mess you up when you write the queries if you don't have exactly the same names. So we want to make sure that we don't waste time. So we're going to call it autocomplete resolver. The pattern type is query.

JASON: Okay.

NATALIA: We're going to create a function that is serverless, autocomplete resolver as well.

JASON: Okay.

NATALIA: The input type in this case is going to be scalar type. String. What we want to get back is the names. And the payload is the existing type, list, and artist. Perfect. Save it. And we are going to write our first aggregation. In fact, we're not going to write it. We're going to copy it, but we're going to explain it.

JASON: Okay. So let's go back here, aggregation.

NATALIA: No, it's not aggregation. It's the resolvers. There should be one called custom resolvers.

JASON: Here?

NATALIA: Yes, that's right. And autocomplete. That's right.

JASON: Okay.

NATALIA: So we are going to copy this and explain this. As you can see, it's in exports. It's in an export of a function. Edit. No, this is a resolver, sorry. Go to functions on the left side bar. GraphQL right below. Exactly. So we have -- let's go over this. We have a function that is a name, right. And we are also querying our context. Our context service is MongoDB Atlas for everything that's in Realm. Our database is tattoo network, our collection is artists. And once we have this query, we're going to aggregate -- like, if you do this in the Mongo shell, you do it in exactly the same way. You have to query the database, the collection, and then pass the aggregate. Into the aggregate, we pass an array of selectors. In this case, composing an aggregation pipeline in the form of stages. Our first stage, it's the search selector. So this dollar search, it's a function, and it's going to be using the index names that we just created with an autocomplete for the path or field name. It's going to query the parameter we pass on top, which is name. It's going to also allow us to do fuzzy search. Fuzzy is when you type in the autocomplete and mistype and it gets it fixed for you. This is what's called fuzzy, with a maximum edit of one. So only one misspelling is going to correct for you.

JASON: Got it.

NATALIA: Then the next thing it does in the next stage is a project. You get a lot of stuff back from the database. This collection has a lot of fields, and you want to optimize. So you're saying the ID is always mandatory. It's always returned. You don't want it. So you are disabling it. You just want to get the name. When you do a project and say I don't want the ID, you have to enable any other field you want. So you have to explicitly say what you want next.

JASON: Got it. So to clarify what that means, we're doing a search, and then when you have a big collection of data, a term that I think is what we're using here is if you want to look at a subset of your data, you're creating a projection of a little bit of data, right? Am I using that word correctly?

NATALIA: You're actually searching the whole data, but because it's indexed, you already indexed this query, so it's going to be a lot more optimal than if you have an uncovered query, which is when you're querying data that's not indexed. So you have everything -- you have all these names in your memory, basically. Right. So it doesn't have to go all the way through the database and scan and look for the right one. No, it's in the memory. It knows already what it has to look for, which is the names field. The project is what you get back. So you get only the name. You don't get the whole document. You don't get all the fields.

JASON: Gotcha.

NATALIA: Then you're limiting to ten because, yeah, that's a way to do pagination. We're not going to implement the full pagination. If you did, this is the way you do it. So you can test it. You go to the console, like over there. Instead of hello world, you can write I think McDonald. It's one of the names in the database.

JASON: Like that?

NATALIA: And McDonald. But actually, you can query don. And just run and you should get the results.

JASON: There we go. There's Donald McDonald. We've got a whole set. That was nice and fast.

NATALIA: Exactly.

JASON: This is great.

NATALIA: So we can save the draft, review and deploy.

JASON: Okay.

NATALIA: All right. And now, yes, I will want you to do a little thing to make sure that we don't have errors. Just go to the repo.

JASON: Yeah, sorry, sorry. Yep.

NATALIA: So go to the top folder. You should see schema. Yeah. There should be one called special case. Just copy these, please. It's the artists, but we're also adding highlights. Highlights is when you get hits on a word. It returns a special object that tells you what is the value that you are looking for and where it found the hit. So copy this. Because the query later includes this, and I don't want to get errors because we typed so much. And we go back to GraphQL. Sorry, to rules. I apologize. Data access rules. We go back to our artist collection. And for the schema, replace this.

JASON: Just replace it?

NATALIA: Yes. And save it.

JASON: Okay. Do you want me to deploy this?

NATALIA: We can deploy it together maybe. We go back to GraphQL. Custom resolver, we're going to create one more. It's going to be called all artists resolver.

JASON: Okay. That's the one we saw back here. Custom resolvers. All artist resolver.

NATALIA: So we have the input type that is a special one. Maybe you want to copy this.

JASON: I've got it.

NATALIA: Yeah, the parent type is query. Then we create a function called the same. Yeah, all artist resolver. Then we are going to add --

JASON: So, I think I need this part first. Right? The function.

NATALIA: Yeah.

JASON: Okay. So there's our function. Here's our input type. And we're going to do a custom type?

NATALIA: Exactly. We're going to add a custom type because we want to pass more than just a scalar type that is the default. So we want to pass all artist input, which is the term, the facets, the country, and the rating.

JASON: Got it.

NATALIA: You want an aggregation of all that. And this is artists.

JASON: Okay.

NATALIA: And we save it.

JASON: Saved.

NATALIA: And we can go, actually, to the GraphQL explorer.

JASON: Do I need to deploy before?

NATALIA: Yeah, I think it will work anyway, but let's do it.

JASON: Okay. So we should see here some changes now, where we've got --

NATALIA: Yes, you should see the schema has changed.

JASON: Great. So there we go. Now I'm going to go to explore. And I can --

NATALIA: You can copy the query if you want. It's all artist resolver. Yes. It's going to be expecting an input that is term facets. In this case, the term is the full name. It's not going to do autocomplete for you. Anyway, it may find it. But let's put Donald McDonald. And facet would be the styles. It's an array of strings.

JASON: Okay. Do you -- let me copy this. I feel like there's going to be a lot of --

NATALIA: I think I'm not passing anything.

JASON: Okay. So you're doing -- I'm just going to grab the one that you did and we'll get that. So let's drop this one in. And we get all the Daniels.

NATALIA: But also -- yeah, all the Daniels and the Danielles, right.

JASON: Yeah.

NATALIA: So it's running a full-text search. And this is why it's returning everything that matches.

JASON: Now, if I simplify this and say we also only want to get trash polka, which I really want to know what that tattoo style looks like, if we had more time, I would be Googling that right now.

NATALIA: Yeah, so you see you get all the trash polka. And you can Google it right now. This is how you do all these amazing aggregations. I would like you to go -- because anyway, we are running out of time. So we are not going to be able to see the full act. If you have time, I would love to come back and finish this.

JASON: Yeah, we can talk about how to get this finished up. We've got about five-ish more minutes to go if we want to try to get maybe a step further.

NATALIA: Yeah, yeah. Let's go ahead. I just want to really talk about the function that powers this. This is the aggregation framework for MongoDB. As you can see, we have again the search stage, the search selector. And we are passing a compound. We say, okay, if we have a term, it must match to this term. Then again, if we have some -- if you go below, you see that we are doing some logic down there. More, more, more. So if you have a term, let's push this into the compound. It should match this term. And if it has facets, then it must match this. And should is -- sorry, must is not as restrictive as should. So it may or may not. But should, it should.

JASON: So, okay. That's actually a question that came in. When we added trash polka as a facet, we started seeing people whose name didn't match Daniel, but their style did match trash polka. So is the way this is set up, the must, it pulled -- so the must pulled in everything that was trash polka, and the should says, if it matches, highlight that. If it doesn't, that's okay. Still show results that match the rest of the search.

NATALIA: Exactly.

JASON: Got it, got it, got it.

NATALIA: That's correct. And then we have also the country, which is also a must. It must be the country. This is basically it. We're also doing a protection in this case. So we guarantee that we're not getting more data than we would need back from the database in this case.

JASON: Cool.

NATALIA: And there was one last custom resolver. I don't know how you want to proceed because actually, if we don't have all of this in place, the application won't compile.

JASON: Let's go fast then. So I'll put in a custom resolver. This is the all opinions resolver.

NATALIA: Correct.

JASON: So I'm going to copy this. We're going to run over and do a -- how do I get back out? What have I done? Oh, I have a collapser. All right. So let's go to our GraphQL custom resolvers. This one is going to be called all opinions resolver. That is going to be a new function. That is also all opinions resolver.

NATALIA: That's right.

JASON: It has this content. So this looks fairly familiar. We're doing a search. We're doing a projection. Good.

NATALIA: Exactly, exactly.

JASON: The input type is?

NATALIA: The input type is ID. So scalar type. Integer, sorry. It's going to be the ID. And it's going to return opinions.

JASON: Got it.

NATALIA: Changes could not be saved.

JASON: You must select a parent type. That's my bad. Okay. Saving. All right. And then do I need to add other things for this to work?

NATALIA: I am looking at it. It may compile now. Let's try it. Go to the app. The thing is you're going to have to install it, right. It's an Angular application. It takes a bit. Maybe we should have done this before.

JASON: Uh, what happened? Oh, there's no package JSON.

NATALIA: What? Okay. Let me see if I can push it. That's weird. There is none?

JASON: I don't see one.

NATALIA: Package JSON. I do see it.

JASON: It must be ignored or something.

NATALIA: It doesn't look like it is.

JASON: Is it inside a folder? Do I need to go in -- Angular tattoo app. Oh, I need to go into the app, don't I?

NATALIA: Oh, okay. Sorry, sorry. I didn't realize. I was like, okay, weird. I see it.

JASON: Okay. So I'm in the app now. And we're running an install. And I have pretty fast internet, so hopefully this'll happen quickly. We're in a race against time here. While we're waiting for this to install, where should people go if they want to learn more about MongoDB?

NATALIA: Okay. There are several places. They should always -- and I always say this -- go to the official documentation, MongoDB.com, documentation. So, resources. Documentation. Always go to the official documentation. You can always also go to the MongoDB University, which is free as well and has tons of courses that even give you a nice certificate. You just sign up and choose your pass. You want to be a developer, you want to be a DBA. You can choose your path. And it's going to teach you a lot of things. You can even certify later, if you so wish.

JASON: Great. So this looks like it --

NATALIA: It's good?

JASON: Let's try it.

NATALIA: Okay. Ah, there are some things you're going to have to set up first and cannot do it from -- yeah. So let's do this. You have to go to the app in Realm and copy the ID. Do you remember I told you do not expose this?

JASON: Mm-hmm.

NATALIA: Okay. And you're going to have to replace it in the GraphQL module.

JASON: And that is going to be in the app.

NATALIA: That's right.

JASON: GraphQL module. Your app ID. So I don't want to expose this?

NATALIA: Yeah, maybe you don't want to expose this.

JASON: Okay. That's okay. Then I'm going to copy out from GraphQL. It looks like I also need the --

NATALIA: URL, which you see there.

JASON: Let me make this bigger so I can actually copy it. There we go. All right. So I am saving both of those things. I'm going to bring this back over now that I'm hidden. So I've got my API and the next line up is the key there. So I'm going to close that. Let's stop, recompile. And we've got four minutes. So while we're waiting for this to recompile --

NATALIA: Run, run!

NATALIA: I'm going to do a couple things here. Let's first do a shout out to our live captioning. We have had Rachel here all day from White Coat Captioning doing the captioning for us. Thank you very much, Rachel.

NATALIA: Thank you.

JASON: And that was made possible through the support of our sponsors. We have Netlify, Fauna, and Auth0 all kicking in to make this show more accessible. If you enjoyed the show today, you can always find the schedule, which -- oh, no. I never built that, like, you've changed the window size thing. So you can go find the schedule. We've got a ton of great content coming up. We're going to do passwordless auth. We're going to do a UI/UX diagnosis and rebuild, which is going to be really, really fun. We've got n8n automation. We have Ben Holmes, who might be in the chat today, who's going to teach us about islands architecture. So hit that add on Google Calendar and follow me on Twitch so you see those episodes as they happen.

NATALIA: So cool.

JASON: We're compiled successfully. That means, theoretically, when I open this, we have tattoos. Fingers crossed, everybody.

NATALIA: Oh, I don't see anything. Now I'm nervous.

JASON: Oh, it immediately failed. We're missing something. Refresh custom data.

NATALIA: Okay. Refresh custom data is a method -- what it does is it uses your -- oh, there is an ID. Ah, I know what it is. Okay. So what we didn't do is run back to Realm. Let's go fix that.

JASON: Oh, boy. Two minutes. Here we go.

NATALIA: Where you were. You were already there. So authentication. Data access. So allow users to log in anonymously. Switch on. And save it. And deploy it.

JASON: And recompile?

NATALIA: Yes. And if it doesn't work, I want to continue. This is so much fun. Also, because the fun part was also that we're going to create a static side with all these profiles. And we're going to deploy them to Netlify every time we make changes to the database with an incoming hook.

JASON: Okay. So now we've got, let's see. Still doesn't like this refresh custom data. So I'm not 100% sure what's going on. We'll have to debug that at another time. However, we were able to do quite a bit here. So we were able to get in -- oh, no. I didn't finish deploying. That's why that didn't work. Dang it. Okay. This try is going to work. Everybody get ready. Okay. We have exactly one minute, which means this is the moment of truth.

NATALIA: Let's see.

JASON: Do the thing! Do the thing! Do the thing! Okay. So if this one doesn't work, we were able to set up a database in MongoDB Atlas. We were able to load custom data. We got the CLI installed. If you have home brew, that's going to be a lot less painful than it was for me. We were then able to get the data in, create custom indexes, hook that up to GraphQL through Realm, and then set up some custom resolvers so we have autocomplete and a whole bunch of really good stuff going on in there. This is pretty powerful stuff. Okay. So it still doesn't like this refresh custom data, which means we probably missed a step.

NATALIA: Boo-hoo.

JASON: That's okay. We'll get this -- building the front end wasn't really the part we were excited about here. The part we were excited about was getting that data accessible. We have that GraphQL database we can use. We'll make come back and build the front end on top of this app we built. And with that, we're out of time. Natalia, any parting words before we call this one done?

NATALIA: Yeah, thank you very much. And I hope that everyone goes to MongoDB University, learns more. There is a lot of powerful stuff they can use to build applications without having to know about the underlying infrastructure about how things get really -- you know, makes sense underneath. Thank you very much for inviting me. Looking forward to a follow-up. Yeah, that's it.

JASON: Me too. All right, y'all. Thank you so much for hanging out with us. Stay tuned, chat. We're going to find somebody to raid. We will see you all next time.

NATALIA: Bye-bye.