Skip to main content

Posts tagged with 'FTS'

This is a repost that originally appeared on the Couchbase Blog: Geospatial Search with ASP.NET Core, Aurelia, and Google Maps.

Geospatial search is now fully supported in Couchbase Server 5.5. Check out the Couchbase Server 5.5 announcement, and download the developer build for free right now.

In this post, I’m going to demonstrate the geospatial search capabilities of Couchbase Full Text Search by creating a web-based UI that performs searches. Whenever I think of geospatial searches, I think about Yelp, which is great at helping me find restaurants in a specific area.

So I’m going to have a little fun and create a very bare-bones version of Yelp, but only for hotels.

If you want to follow along, the full source code is available on Github.

Getting set up

Here are the steps I took to create a new project before I started writing code.

  1. At the command line: dotnet new aurelia. This assumes that you have .NET Core installed. Note that Geospatial Search is not a .NET-only feature: you can use it with the other Couchbase SDKs like Node.js, Java, etc. It also assumes that you’ve installed a SPA template for Aurelia. You can also go with Angular or React if you’d like, but I really like Aurelia, and I think you should give it a chance.

  2. The above command will create a shell of an ASP.NET Core project. In this blog post, I’m not going to use Razor. I’m just using ASP.NET as a backend for REST API endpoints.

  3. npm install aurelia-google-maps. You don’t have to use this, but the aurelia-google-maps plugin will make it easy for me to interact with Google Maps in my app.

  4. I opened this project in Visual Studio 2017. I added Couchbase.Extensions.DependencyInjection with NuGet. You don’t have to use this extension but it makes things easier.

  5. I installed Couchbase Server 5.5, including the Full Text Search service. I setup the travel-sample bucket. I created a user "matt" with full access to that bucket.

Create a Geospatial Index

Before building the ASP.NET backend, we need to create a geospatial index in Couchbase Server. Once you log in, click "Search" on the menu (it’s under "Workbench"). Click "Add Index" to get started.

Create Geospatial index

I named my index "mygeoindex". I selected travel-sample as the bucket to index.

In "Type Mappings", I uncheck the default. I add a new type mapping with a type name of "hotel". Every hotel document in "travel-sample" has a type with a value of "hotel". Check the "only index specified fields" box.

I’m going to add two child fields. One is "geo", which contains the geospatial coordinates inside a hotel document. Make sure to select "geopoint" as the type. The other is "name", which will be the name of the hotel. I choose to "store" each of these: it will make the index larger, but I can avoid a secondary lookup if I store the information in the index.

Important Note: There is a bug (NCBC-1651) in the current release of the .NET SDK that will cause an error if you try to read from a geopoint field. In the code samples, I’ve created a workaround: I don’t actually get the geo & name fields from the search index. I instead use the document key returned by search to make a secondary "get" call and get the full document. Keep in mind this is still a technique you may want to consider if you want to keep the size of your index down. This bug has already been fixed and will be in a future release. Such is the peril of being on the cutting edge!

That’s all there is to it. Click "Create Index". Watch the "indexing progress" on the next screen until it gets to 100% (it should not take very long, assuming you remembered to uncheck "default").

ASP.NET Core REST Endpoints

Next, let’s move over to ASP.NET. I’ll create two endpoints. One endpoint will demonstrate the bounding box search method, and the other will demonstrate the distance search method.

I’ll need a Couchbase bucket object to execute the queries. Follow the examples in my blog post about dependency injection or check out the source code on Github if you’ve never done this before.

Bounding Box

A "bounding box" search means that you define a box on a map, and you want to search for points of interest that are inside of that box. You only need two points to define a box: the top right corner coordinates and the bottom left corner coordinates. (Coordinates are latitude and longitude).

public class BoxSearch
{
    public double LatitudeTopLeft { get; set; }
    public double LongitudeTopLeft { get; set; }
    public double LatitudeBottomRight { get; set; }
    public double LongitudeBottomRight { get; set; }
}

To create a bounding box geospatial query, use the GeoBoundingBoxQuery class available in the .NET SDK. I’ll do this inside of a POST method with the above BoxSearch class as a parameter.

        [Route("api/Box")]
        [HttpPost]
        public IActionResult Box([FromBody] BoxSearch box)
        {
            var query = new GeoBoundingBoxQuery();
            query.TopLeft(box.LongitudeTopLeft, box.LatitudeTopLeft);
            query.BottomRight(box.LongitudeBottomRight, box.LatitudeBottomRight);
            var searchParams = new SearchParams()
                // .Fields("geo", "name") // omitting because of bug NCBC-1651
                .Limit(10)
                .Timeout(TimeSpan.FromMilliseconds(10000));
            var searchQuery = new SearchQuery
            {
                Query = query,
                Index = "mygeoindex",
                SearchParams = searchParams
            };
            var results = _bucket.Query(searchQuery);

// ... snip ...

All I need to return from this endpoint is a list of the results: each hotel’s coordinates and the hotel’s name & location. I created a GeoSearchResult class for this.

public class GeoSearchResult
{
    public double Latitude { get; set; }
    public double Longitude { get; set; }
    public InfoWindow InfoWindow { get; set; }
}

public class InfoWindow
{
    public string Content { get; set; }
}

I’ve constructed this class to match the Google Maps plugin that I’ll be using later.

Finally, I’ll use this class to return some results from the endpoint.

// ... snip ...

            var list = new List<GeoSearchResult>();
            foreach (var hit in results.Hits)
            {
                // *** this part shouldn't be necessary
                // the geo and name should come with the search results
                // but there's an SDK bug NCBC-1651
                var doc = _bucket.Get<dynamic>(hit.Id).Value;
                // ****************
                list.Add(new GeoSearchResult
                {
                    Latitude = doc.geo.lat,
                    Longitude = doc.geo.lon,
                    InfoWindow = new InfoWindow
                    {
                        Content = doc.name + "<br />" +
                            doc.city + ", " +
                            doc.state + " " +
                            doc.country
                    }
                });
            }
            return Ok(list);
        }

A "distance" search is another way to perform geospatial queries. This time, instead of a box, it will be more like a circle. You supply a single coordinate, and a distance. The distance will be the radius from that point.

public class PointSearch
{
    public double Latitude { get; set; }
    public double Longitude { get; set; }
    public int Distance { get; set; }
    // miles is being assumed as the unit
    public string DistanceWithUnits => Distance + "mi";
}

I’m defaulting it to miles, but certainly you can use kilometers instead, or present the option in the UI.

The endpoint will be very similar to the bounding box endpoint, except that it uses GeoDistanceQuery.

[Route("api/Point")]
[HttpPost]
public IActionResult Point([FromBody] PointSearch point)
{
    var query = new GeoDistanceQuery();
    query.Latitude(point.Latitude);
    query.Longitude(point.Longitude);
    query.Distance(point.DistanceWithUnits);
    var searchParams = new SearchParams()
        // .Fields("geo", "name") // omitting because of bug NCBC-1651
        .Limit(10)
        .Timeout(TimeSpan.FromMilliseconds(10000));
    var searchQuery = new SearchQuery
    {
        Query = query,
        Index = "mygeoindex",
        SearchParams = searchParams
    };
    var results = _bucket.Query(searchQuery);

    var list = new List<GeoSearchResult>();
    foreach (var hit in results.Hits)
    {
        // *** this part shouldn't be necessary
        // the geo and name should come with the search results
        // but there's an SDK bug NCBC-1651
        var doc = _bucket.Get<dynamic>(hit.Id).Value;
        // ****************
        list.Add(new GeoSearchResult
        {
            Latitude = doc.geo.lat,
            Longitude = doc.geo.lon,
            InfoWindow = new InfoWindow
            {
                Content = doc.name + "<br />" +
                          doc.city + ", " +
                          doc.state + " " +
                          doc.country
            }
        });
    }
    return Ok(list);
}

At this point, you can start testing these endpoint with Postman or Fiddler if you’d like. But it will be so much nice to see this on a map.

Auerlia and Google Maps

In Aurelia, I’ve created two components: geosearchbox and geosearchpoint.

Auerlia components

Each of them will have a Google Maps component that the user can interact with. These maps will be centered on San Francisco, because that’s where a lot of the hotels in "travel-sample" are located.

Bounding Box search component

The google-map` component has a map-click.delegate that will will fire whenever the users clicks on the map. In geosearchbox.html:

<google-map
    if.bind="markers"
    map-click.delegate="clickMap($event)"
    latitude="37.780986253433895"
    longitude="-122.45291600632277"
    zoom="12"
    markers.bind="markers">
</google-map>

markers is simply an array containing coordinates of search results that should appear on the map. Initially it will be empty.

When the user first clicks the map, this will set the first coordinate (top left) in the form. In geosearchbox.ts:

public clickMap(event : any) {
    var latLng = event.detail.latLng,
        lat = latLng.lat(),
        lng = latLng.lng();

    // only update top left if it hasn't been set yet
    // or if bottom right is already set
    if (!this.longitudeTopLeft || this.longitudeBottomRight) {
        this.longitudeTopLeft = lng;
        this.latitudeTopLeft = lat;
        this.longitudeBottomRight = null;
        this.latitudeBottomRight = null;
    } else {
        this.longitudeBottomRight = lng;
        this.latitudeBottomRight = lat;
    }
}

Then, click another spot on the map. This will set the second coordinate (bottom right).

My implementation is very bare bones. No fancy graphics and no validation of the second coordinate being to the bottom right of the first. The fields on a form will simply be populated with the latitude and longitude. In geosearchbox.html:

<p>
    Bounding box search:
    <br />
    Latitude (top left):
        <input type="text" value="${ latitudeTopLeft }" />
    Longitude (top left):
        <input type="text" value="${ longitudeTopLeft }" />
    <br />
    Latitude (bottom right):
        <input type="text" value="${ latitudeBottomRight }" />
    Longitude (bottom right):
        <input type="text" value="${ longitudeBottomRight }" />
    <br />
    <input
        if.bind="latitudeTopLeft && latitudeBottomRight"
        click.trigger="searchClick()"
        type="button"
        name="search"
        value="Search" />
</p>

Once you’ve selected two coordinates, a search button will appear. Click that to post these coordinates to the endpoint created earlier, and it will trigger the searchClick() method as seen in geosearchbox.ts:

public searchClick() {
    let boxSearch = {
        latitudeTopLeft: this.latitudeTopLeft,
        longitudeTopLeft: this.longitudeTopLeft,
        latitudeBottomRight: this.latitudeBottomRight,
        longitudeBottomRight: this.longitudeBottomRight
    };

    console.log("POSTing to api/Box: " + JSON.stringify(boxSearch));

    this.http.fetch('api/Box', { method: "POST", body: json(boxSearch) })
        .then(result => result.json() as Promise<any[]>)
        .then(data => {
            this.markers = data;
        });
}

When Aurelia, Google Maps, ASP.NET Core, and Couchbase all work together, it looks like this:

Geospatial bounding box

Distance Search

Implementing the "distance" geostatial query will be similar to the bounding box UI. This time, you only need to click a single point on the map. But, you will need to type in a distance (in miles).

The google-map component will look identical. The clickMap function is different:

public clickMap(event: any) {
    var latLng = event.detail.latLng,
        lat = latLng.lat(),
        lng = latLng.lng();

    this.longitude = lng;
    this.latitude = lat;
}

Specify a distance (in miles), and then click 'search' to make a POST request to the endpoint we wrote earlier.

geosearchbox.html:

    <p>
        Distance search:
        <br />
        Latitude: <input type="text" value="${ latitude }" />
        Longitude: <input type="text" value="${ longitude }" />
        <br />
        Distance (miles): <input type="text" value="${ distance }" />
        <br />
        <input if.bind="latitude" click.trigger="searchClick()" type="button" name="search" value="Search" />
    </p>

geosearchbox.ts:

    public searchClick() {
        let pointSearch = {
            latitude: this.latitude,
            longitude: this.longitude,
            distance: this.distance
        };

        console.log("POSTing to api/Point: " + JSON.stringify(pointSearch));

        this.http.fetch('api/Point', { method: "POST", body: json(pointSearch) })
            .then(result => result.json() as Promise<any[]>)
            .then(data => {
                this.markers = data;
            });
    }
}

Below is a clip of the search in motion. Note how the results change as I move the coordinate around.

Geospatial distance point search query

Summary

With Couchbase’s built-in geospatial indexing and search feature, all the math and the searching is delegated to the Couchbase Data Platform. So you can focus on building a killer UI (better than mine anyway) and rock-solid business logic.

Be sure to check out the documentation for a complete overview of the geospatial capabilities of Couchbase.

If you need help or have questions, please check out the Couchbase Server forums, and if you have any questions about the Couchbase .NET SDK, check out the .NET SDK forums.

If you’d like to get in touch with me, please leave a comment or find me on Twitter @mgroves.

This is a repost that originally appeared on the Couchbase Blog: Powershell with the Couchbase REST API.

PowerShell is a scripting environment / command line that comes with Windows and is also available for Linux and within Azure.

Maybe you’ve used Postman or Fiddler to make HTTP requests. Those are great, but not necessarily the right tools for automation or scripting.

You may have heard of curl before. It’s a command line tool for making HTTP requests.

If you’re a .NET/Windows developer (like me), maybe you aren’t familiar with curl. I use PowerShell as my default command line every day (though I still consider myself a PowerShell neophyte). In this post, I’ll show you how you can use PowerShell’s Invoke-WebRequest to make HTTP requests (which you can use within PowerShell scripts for automation).

You can check out the PowerShell script I created on GitHub. Note: as of the time of writing this post, I’m using PowerShell 5.1 on Windows 10.

Couchbase REST API

Couchbase Server has a an extensive REST API that you can use to manage and administrate just about every aspect of Couchbase Server. For this blog post, I’m going to focus on the Full Text Search (FTS) API. I’m going to show this because:

  • Creating an FTS index is something you’ll eventually want to automate

  • You will probably want to share an FTS index you created with your team and/or check it into source control

  • Couchbase Console already shows you exactly how to do it with curl.

I’m not going to cover FTS in detail: I invite you to check out past blog posts on FTS, and this short video demonstrating full text search.

Full Text Search review

When you initially create an FTS index, you will probably use the built-in FTS UI in the Couchbase Console. This is fine when you are doing the initial development, but it’s not practical if you want to share this index with your team, automate deployment, or take advantage of source control.

Fortunately, you can use the "Show index definition JSON" feature to see the JSON data that makes up the index definition. You can also have Couchbase Console generate the curl method for you.

Generate FTS curl script

Well, if you’re using curl, that’s very convenient. Here’s an example:

curl -XPUT -H "Content-Type: application/json" http://localhost:8094/api/index/medical-condition -d '{ ... json payload ...}'

You can copy/paste that into a script, and check the script into source control. But what if you don’t use curl?

PowerShell version: Invoke-WebRequest

First, create a new PowerShell script. I called mine createFtsIndex.ps1. All this PowerShell script is going to do is create an FTS index on an existing bucket.

You can start by pasting the "curl" command into this file. The bulk of this command is the JSON definition, which will be exactly the same.

Let’s breakdown the rest of the curl command to see what’s happening:

  • -XPUT - This is telling curl to use the PUT verb with the HTTP request

  • -H "Content-Type: application/json" - Use a Content-Type header.

  • http://localhost:8094/api/index/medical-condition - This is the URL of the REST endpoint. The "localhost" will vary based on where Couchbase is running, and the "medical-condition" part is just the name of the FTS index.

  • -d '…​json payload…​' - The body of content that will be included in the HTTP request.

PowerShell’s Invoke-WebRequest can do all this stuff too, but the syntax is a bit different. Let’s step through the equivalents:

  • -Method PUT - This is telling Invoke-WebRequest to use the PUT verb with the HTTP request, so you can replace -XPUT

  • -Header @{ …​ } - Specify headers to use with the request (more on this later)

  • -Uri http://localhost:8094/api/index/medical-condition" - You just need to add "-Uri" in front

  • -Body '…​json payload…​' - The body of content is included this way instead of using curl’s -d

Headers

PowerShell expects a "dictionary" that contains headers. The syntax for a literal dictionary in PowerShell is:

@{"key1"="value1"; "key2"="value2"}

So then, to specify Content-Type:

-Headers @{"Content-Type"="application/json"}

One thing that the curl output did not generate is the authentication information that you need to make a request to the API. With curl, you can specify basic authentication by adding the username/password to the URL. It will then translate it into the appropriate Basic Auth headers.

With PowerShell, it appears you have to do that yourself. My local Couchbase Server has credentials "Administrator" and "password" (please don’t use those in production). Those need to be encoded into Base64 and added to the headers.

Then the full Headers dictionary looks like this:

-Headers @{"Authorization" = "Basic "+[System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes("Administrator:password")); "Content-Type"="application/json"}

You might think that’s a bit noisy, and I agree. If you know a cleaner way to do this, I’m dying to know. Please leave a comment.

Execute the PowerShell script

To execute the script, simply type .\createFtsIndex.ps1 at the PowerShell command line.

Execute PowerShell script

You’re now ready to make this a part of your deployment.

Summary

PowerShell is a handy tool for scripting and automation. It’s used by Azure, Octopus Deploy, and anywhere Windows is running. Use PowerShell to automate Couchbase REST API calls for things like Full Text Search indexes, and your team will thank you.

Need help? Reach out to me with questions by leaving a comment below or finding me on Twitter @mgroves.

Matthew D. Groves

About the Author

Matthew D. Groves lives in Central Ohio. He works remotely, loves to code, and is a Microsoft MVP.

Latest Comments

Twitter