rclip 3 is out. It improves search quality and makes search up to 6x faster.

rclip is a command-line tool for searching local photos by text or by example image. If you’re new to it, start with the intro post here.

The major changes that shipped with this version are:

• rclip now uses OpenCLIP’s top-performing ViT-B/32 model, which improves benchmark accuracy. On ImageNet-1k, rclip 2 reached 58.78% top-1 accuracy and 84.21% top-5 accuracy, while rclip 3 reaches 68.73% top-1 accuracy and 89.59% top-5 accuracy.
• rclip now searches up to 6x faster (depends on your OS and hardware). For example, text search on M1 Max went from 2.12 seconds to 0.57 seconds (3.72x speedup and near-instant feel), while image-to-image search on the same machine went from 3.84 seconds to 0.63 seconds (a 6.09x speedup!).

Real-world gains will vary by photo collection.

Before I talk about the implementation details, I invite you to install the latest version. Follow the installation instructions from the README to do so. Since rclip now uses a new model, it’ll have to reindex your images.

Under the hood, these gains came from two independent changes.

The model change was long overdue. The original OpenAI CLIP model was no longer the best default choice, and the benchmark gains above made the switch worth it.

The runtime change was enabled by replacing the large and slow-to-load PyTorch with onnxruntime, which loads very fast, packages better, and provides very similar performance to PyTorch when run on a CPU. Most of the search speed gains came from getting rid of long PyTorch loading times.

The only caveat with the switch to onnxruntime was that Apple Silicon support in PyTorch is better during batch processing, so indexing speeds on Apple Silicon regressed. To fix this, rclip 3 for macOS ships with one more execution backend: coremltools, which is provided by Apple and optimized for macOS. coremltools has the same slow startup downside as PyTorch, so rclip 3 uses it on macOS only for indexing, while defaulting to onnxruntime for search. Thanks to the coremltools optimizations, indexing speeds on macOS went up. In my tests, I got 180 images per second, which is 12.5% faster than the 160 images per second I had been getting with PyTorch.

These improvements also come in a smaller package. For example, the rclip Windows MSI installer went from 172 MB in 2.1.6 to 59 MB in 3.0.9. rclip snap went from 224 MB to 57 MB. I realize that these days, a lot of people don’t care about download sizes, but it’s hard not to appreciate the software that needs less: less compute, less space. Also, as a person obsessed with great UX and responsive apps, I am really happy that downloading and installing rclip got faster, too.

If you’ve been looking for a great text-to-image and image-to-image semantic photo search that lets you mix images and text queries and preview image search results in the terminal, it just got faster. Install rclip, let it reindex with the new model, and let me know what you think.

Stay curious!

This week, I released version 2 of rclip. The main breaking change in the major update is that rclip dropped Python 3.9 support, which could affect you only if you are installing rclip using pip in a Python 3.9 environment. Even though no features are associated with the v2 release, I am thrilled about it because it provides a good opportunity to reflect on the changes made to rclip since the release of version 1.

For those unfamiliar with rclip, it is a command line tool that allows you to search your image or photo collection right in your terminal using a natural language query or a reference image. To try rclip, follow the installation instructions on the GitHub page.

I have already blogged about most of the major features introduced to rclip v1:

• image-to-image search
• composite queries, including mixed text and image queries
• terminal image previews

What I didn’t blog about is the countless performance improvements, from a few noticeable indexation speedups to a 50% search time reduction when performing text-only queries.

I am also excited to announce for the first time that rclip now has experimental support for raw image search. GitHub user @abidkhan484 added this feature to rclip during the last Hacktoberfest (speaking of which, rclip will participate in Hacktoberfest 2025 as well – join us!). The feature was first released in rclip v1.11.0, allowing you to search through raw camera images. Currently, only the ARW and CR2 images are supported. Let me know if you would love rclip to support any other raw formats!

To use raw search, run rclip with the --experimental-raw-support flag, like this:

rclip --experimental-raw-support "orange cat"

With this flag passed, rclip will search through regular and raw images and surface best matches. Terminal previews are also supported for raw photos. It is important to note that rclip will still prefer non-raw versions of the image when they are available. Suppose it finds two versions of the same image in the same directory – for example, IMG_1234.JPG and IMG_1234.ARW – it will ignore the raw version.

Install rclip, try this feature, and let me know if you have any feedback. Stay curious, and enjoy your photo searches with rclip!

Hi everyone!

I’m thrilled to share a long-overdue update about an exciting rclip feature. Now, you can preview search results directly in the terminal!

If you’re new to rclip, it’s an AI-powered command-line photo search tool – think “rgrep” for images. Dive into my previous posts and give it a spin to learn more.

Now, let’s talk about the new preview feature. To witness the magic, simply add the --preview (or -p) argument to your rclip command, like this: rclip -p kitty. Check out the video below to see it in action.

Some might argue that the ability to preview images was in rclip before because you can pipe filenames output by rclip to any other command. You might remember reading a suggestion about piping filenames to an image viewer in rclip’s README. It worked, but the command was a bit of a monstrosity. You had to remember the -f argument, deal with xargs intricacies, and know how to call the viewer you want to use.

The other issue with that suggestion is that it doesn’t work over SSH. This is especially important to me because I do a lot of searches on a remote machine. And, even if you make it work over SSH somehow, it will be slow because it has to transmit the whole image over the network to display a preview unless you go further and compress it yourself.

Now, these problems are no more. I’ve addressed these issues with the new preview feature! No more cumbersome commands. Now, it’s as simple as typing rclip -p kitty, and voila – compressed image previews right in your terminal. It also works seamlessly over SSH. No more sluggish previews; just a smooth, optimized experience, even on remote machines.

The feature is compatible with iTerm2, Konsole (version 22.04 and higher), wezterm, Mintty, and mlterm.

Get the feature by following the instructions in the repository’s README to download the latest version of rclip. Try it out and share your thoughts. Stay curious, and enjoy your photo searches with rclip!

Hi! Today, I am introducing a fantastic new feature of rclip (an AI-powered command-line photo search tool I built): the ability to add and subtract different queries. It’s fun to play around with different concepts in this way. This feature is also handy when you want to find similar images. If it’s the first time you read about rclip, check out this and this blog posts.

The feature works like this. If you want to add two queries together, add a “plus” sign between them like this:

cd photos && rclip horse + stripes

If you want to subtract one query from the other, use the “minus” sign:

cd photos && rclip apple - fruit

It works with image queries too:

cd photos && rclip "./new york city.jpg" + night

When combining image and text queries, you may notice that sometimes your text queries are seemingly ignored by rclip. This can happen because CLIP (an AI model that rclip uses) weights features extracted from images more than features extracted from text. Luckily, rclip provides a way to fix this. You can control the influence the query has on the result by prefixing it with a multiplier. It can be any positive number, and you can multiply both image and text queries. Here is an example which doubles the influence of the “golden retriever” query:

cd photos && rclip "2:golden retriever" + "./swimming pool.jpg"

And last but not least, when combining queries like this, you are not limited to two queries. Combine as many as you want. For example:

cd photos && rclip "./racing car.jpg" - "2:sports car" + "2:snow"

If you are interested in seeing how all these queries perform when executed on the 1.28 million images ImageNet-1k dataset, check out the video demo on YouTube:

This is another rclip feature initially introduced by a GitHub user ramayer. Huge thanks to ramayer for this incredible contribution! Speaking of contributors, I always appreciate the enthusiasm and fresh perspectives that new contributors bring! If you’re considering getting involved in open-source projects and making a positive impact, a number of issues on the rclip GitHub page could benefit from your attention and expertise. Why not give them a look?

That’s all you need to know about the new feature. You can download the latest version of rclip by following the instructions in the repository’s README. Try it out and share your thoughts. Stay curious, and enjoy your photo searches with rclip!

If you have been following my work, you are probably already familiar with rclip, a command-line image search tool I built to solve a specific problem: navigating my personal collection of photos.

Today, I am excited to introduce a new feature to rclip. This update expands the search capabilities of rclip beyond text queries, allowing you to search for images similar to a given image. You only need to provide an image path or URL, and rclip will find visually similar images. For example:

cd photos && rclip ./cat.jpg

# or use URL
cd photos && rclip https://raw.githubusercontent.com/yurijmikhalevich/rclip/main/tests/e2e/images/cat.jpg

This feature was added a while back, but I have only wrapped it up properly recently. It is not just a cool feature; it is special because it was contributed by the first-ever contributor to rclip – a GitHub user ramayer. ramayer reached out with this incredible proposal, worked on the implementation, and after a bit of back-and-forth feedback, I was delighted to merge the PR into the project. A huge shout-out to ramayer for this incredible contribution!

Check out the video below for a demo:

Open source projects like rclip thrive on the energy and innovation of contributors. If you want to dive into open source and make a difference, several issues on the rclip GitHub page could use your attention.

That’s it for now! Test drive this new feature and share your feedback.

Until next time, stay curious!

Meet Minerva, a GPT-powered Discord bot I built to help students in the study group I mentor learn software development and machine learning.

Since 2021, I have been leading a pro-bono study group where I help students from my alma mater develop their skills and learn how to build and ship products. We have a Discord server with channels dedicated to different areas of study and student projects, and we meet face-to-face every week over a video chat. Anything related to any aspect of software development can be discussed there, though we mainly focus on machine learning and web development. In the group, I encourage people to study a lot, make bold mistakes, ask many questions and help each other. But I have noticed that sometimes, our club members hold back from asking “simple” questions, fearing they might needlessly disturb others. Despite understanding this feeling well, I prefer to avoid this frame of mind because even the simplest of questions can provoke an engaging discussion and deepen the knowledge and understanding of everyone involved. I wanted to empower everyone to seek answers without hesitation, so I built Minerva — an AI assistant based on OpenAI’s “gpt-3.5-turbo” model (the same one used in ChatGPT) — to allow members to ask any questions without worrying about disturbing anyone! The bot lives in our Discord server now. To chat with the bot, the members can send a message, as usual, and mention “@Minerva.” Check out this interaction to get a feeling of how it works:

The bot maintains an independent discussion in each server channel, differentiates between messages sent by different club members, knows how to mention Discord users, and formats the code in the responses using markdown! The perfect mentor, Minerva, prefers to nudge the student in the right direction instead of solving a problem for them.

The other purpose of building the bot was to answer a club member’s request to see how the integration with OpenAI’s chat completion model work. I have made this bot for the club members with the club members. Most of the development was done live during our weekly face-to-face meetings. The bot’s source code is a valuable reference for anyone wanting to build their own Discord AI chatbot and is available on GitHub. If you want to let Minerva into your Discord server, follow the instructions in the repository’s README.

I created an avatar for Minerva using OpenAI’s DALL·E and Midjourney. DALL·E was used to iterate over ideas. Midjourney was then used with a hybrid (image and text) prompt to perfect the concept. And for the Minerva banner, I used Canva’s fantastic Magic Design tool.

I have 72,769 photos stored on my NAS and organized into subfolders named according to the pattern “YEAR.MM.DD - Event description.” This organizational approach is software-agnostic and creates a structure that is easy to navigate. I have followed it for more than a dozen years. However, there are situations when I want to find a particular photo. Looking for it even within a specific subfolder can take a lot of time, especially if that subfolder is packed with many pictures. Another challenging scenario is finding a picture that will fit a particular theme without having a specific one in mind. In both of these scenarios, having content-based search available is helpful. I looked around and didn’t find a solution that satisfied me. What to do when you find yourself in a situation like this? Build your own solution!

Techniques allowing you to search images by their content are not necessarily new, but they usually only allow you to search using a set of predefined keywords and, sometimes, their synonyms. OpenAI’s CLIP takes computer image understanding to a whole new level, allowing us to search for photos using any text query we can think of. This is why I used it to build my tool — rclip. In essence, rclip is a simple interface to the CLIP model. All I had to do was load the model, extract feature vectors from all of the images in the directory, store them somewhere, extract the feature vector from the query, compute similarities between the query feature vector and the image feature vectors, and output the most similar images. The vector extraction is done only once for each image, on the first run, and then the vectors are stored in the SQLite DB. SQLite is not the most performant way to manage that kind of data, but it’s a quite convenient one. And, given that the time needed to query the DB on my 73 thousand photos catalog contributes only up to 12% of the whole execution time, I have decided to keep it for the time being.

I use rgrep a lot. rgrep is a simple alias to the grep command-line tool that searches recursively for a substring match in all of the files in the current directory. In its minimal definition (without a bunch of unwanted directories excluded), it looks like this: alias rgrep="grep -rni". To use it, open your terminal, navigate to the directory containing the files that you want to grep, type “rgrep” “search query,” and hit “Enter.”

rclip implements exactly the same interface, but searches through the images’ content. The name stands for “recursive CLIP.” Its usage looks like this:

cd images && rclip "search query"

For the detailed demonstration, watch the video below:

That’s it! If you want to try rclip yourself, follow the installation instructions described here.

This post is a walkthrough of the winning solution for the ML Prague 2021 Hackathon problem that my teammate Michał and I came up with.

We were given anonymized data containing approximately 524k Czech policyholders, 44k clinic departments of 273 different types, and 332 ambulance stations. We were tasked with assigning each of the policyholders to the closest of each of the 273 clinics, in addition to the nearest ambulance station. The closest clinics and ambulance stations were supposed to have been picked by the driving distance. To help us with the routing, we were given $200USD in MS Azure credit, which was enough to make 40k routing requests (or 400k if we were willing to go under the 50 queries per second limit). This amount is nowhere near the one required for a brute-force solution, not to mention that it would take a while. We had five days to come up with the solution. On top of that, we were tasked to present our results using SAS Visual Analytics on SAS Vija.

If you prefer video to text, here is our presentation for the conference featuring the dashboard:

The video is denser and less detailed compared to this article.

Before we dive into the code, let’s go over the solution’s summary.

We’ve noticed that 5,827 of 43,908 clinics lacked data essential for the task: type and department_id. Online research failed to turn up a way to fill in the data, so we dropped the clinics that didn’t have these two fields filled in.

Since ambulance stations and clinics had the same data structure, we introduced a new, special type, 'AST', to identify ambulance stations and concatenated two tables into an “organizations” table.

Then, we’ve created a parent_orgs table that grouped organizations by their geolocation, preserving pairings between type and department_id, and thus, giving us a list of 10,797 unique geopoints we had to match users with.

The straightforward approach was to iterate over all users to find the closest organizations by computing driving distances between a user and the organizations, which were sorted by straight line distances, stopping when the straight line distance to the next candidate is larger than the driving distance to the best candidate.

But this approach proved inefficient even taking the sorting optimization into account; in a second-best case scenario, we had to compute 524102*274 routes. The actual best-case scenario number would be lower because a lot of department_ids share the same coordinate.

Instead, we decided to cluster users, compute driving distances to the centroids of the clusters, and approximate actual driving distances and times for each user based on the relation of the direct_distance(user, organization) / direct_distance(cluster_centroid, organization). The algorithm isn’t perfect but gives us a very close approximation and can be computed in a reasonable time.

We clustered users within areas with a max radius of 500m using the combination of village+village_part to speed up clustering and to limit clusters to a single village_part. This provided us with 47,199 clusters, resulting in much more requests than we were comfortable issuing to MS Azure, considering the provided credit amount. Still, we didn’t want to sacrifice precision, so we set up a local OSRM server and used it to compute driving distances.

For direct distance computation, we used the haversine distance instead of the Vincenty, despite the former being less precise for geodesic computations because we could compute the haversine distance much faster. The error margin it introduces is not significant on the scale that we worked with, and both these distances provide optimistic approximations of the driving distance.

We also noticed that there are many deparment_ids of the same type, sharing the exact location. In these cases, we randomly distributed the corresponding users among them to spread the load between departments.

Alternatively, we could’ve used the fact that some people live on the same street to approximate driving distances, but we believe that small clusters approximate actual distances better.

Possible future improvements:

• Fill in missing specialty_id and department_id.
• Reduce the cluster radius further, allow for more compute time, and re-compute distances with a smaller error margin.
• Consider the distance between an ambulance station and a clinic when assigning an ambulance station to a clinic⇔user/cluster pair and choose an ambulance station closer to the clinic if there are multiple stations available within a similarly close distance to the user/cluster, so that the ambulance car can return to the station ASAP.

Now, let’s dive into the code. The code is in Python and is meant to be executed in a Jupyter Notebook.

According to the task, the output CSV should contain a row for each of the policyholder⇔clinic department pair in the following format:

OUTPUT_COLUMNS = [
    'policyholder_num',
    'contractual_specialty_id',  # clinic department type
    'pzs_department_id',  # clinic department id
    'km_to_pzs',
    'min_to_pzs',
    'zzs_department_id',  # ambulance station department id
    'km_to_zzs',
    'min_to_zzs'
]

Data Exploration

Data exploration is the first thing you should always do when faced with an ML or a data analysis challenge. If you overlook and neglect the importance of this step and do not set aside enough time for it, you may miss important insights, and, consequently, you may fail to implement an optimal solution.

We will use pandas to load and analyze the data.

import pandas as pd

Users (Policyholders)

We start by loading and looking at the policyholders. Let’s call them “users” moving forward.

users = pd.read_csv('./table_POJ.csv')
users.head()

users.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 524102 entries, 0 to 524101
Data columns (total 9 columns):
 #   Column            Non-Null Count   Dtype  
---  ------            --------------   -----  
 0   policyholder_num  524102 non-null  int64  
 1   village           524102 non-null  object 
 2   village_part      524102 non-null  object 
 3   street            289036 non-null  object 
 4   land_num_type     524102 non-null  object 
 5   land_reg_num      524102 non-null  int64  
 6   house_num         118529 non-null  float64
 7   WGS84_N           524102 non-null  float64
 8   WGS84_E           524102 non-null  float64
dtypes: float64(3), int64(2), object(4)
memory usage: 36.0+ MB

There are lots of null values for streets and house numbers. Fortunately, Azure API can compute routes between lat,lon pairs.

users.nunique()

policyholder_num    524102
village               5342
village_part         10706
street               23471
land_num_type            2
land_reg_num          6265
house_num              469
WGS84_N             523864
WGS84_E             523907
dtype: int64

During this stage of the project, it’s essential to understand your data very well.

We can see that villages and village_parts are filled for every user, and they already represent clusters of users, so we may use them for routing optimizations in the future. Also, we can use street info to better approximate driving distances and times for each user.

Clinics

Now let’s look at the clinics.

clinics = pd.read_csv('./table_PZS.csv')
clinics.head()

clinics.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 43908 entries, 0 to 43907
Data columns (total 11 columns):
 #   Column                    Non-Null Count  Dtype  
---  ------                    --------------  -----  
 0   intern_subj_id            43908 non-null  int64  
 1   name                      43908 non-null  object 
 2   village                   43908 non-null  object 
 3   street                    41829 non-null  object 
 4   number                    43908 non-null  int64  
 5   zip                       43908 non-null  int64  
 6   WGS84_E                   43908 non-null  float64
 7   WGS84_N                   43908 non-null  float64
 8   company_id                43908 non-null  int64  
 9   department_id             38081 non-null  float64
 10  contractual_specialty_id  38081 non-null  object 
dtypes: float64(3), int64(4), object(4)
memory usage: 3.7+ MB

clinics.nunique()

intern_subj_id              26322
name                        22816
village                      1722
street                       4442
number                       2966
zip                          1720
WGS84_E                     13893
WGS84_N                     13893
company_id                  26322
department_id               38080
contractual_specialty_id      273
dtype: int64

We see that a lot of clinics share their location. This is not surprising given that each row in the table represents an individual clinic department. But, for convenience, we will continue to call them “clinics.”

Missing streets shouldn’t be a big deal since we can calculate distances using lat,lon, and these are filled in for every clinic.

We also notice that a lot of clinics are missing department_id and contractual_specialty_id. If we drop these clinics, we ignore 6k of them, which is a lot.

Notice that department_id and contractual_specialty_id have the same amount of null values. Let’s check whether all of the clinics missing one are also missing the other.

(clinics['department_id'].isna() == clinics['contractual_specialty_id'].isna()).all()

True

They are. Let’s see what we can do to fill the nulls in. The map of the clinics might give us some insight.

import ipyleaflet as ll
CENTER_OF_CZECHIA = (49.856914, 15.238706)

specialful_clinic_markers = []
specialless_clinic_markers = []

specialful_clinic_icon = ll.AwesomeIcon(name='check', marker_color='green', icon_color='white')
specialless_clinic_icon = ll.AwesomeIcon(name='exclamation', marker_color='red', icon_color='white')

for _, row in clinics.iterrows():
    if pd.isna(row['contractual_specialty_id']):
        specialless_clinic_markers.append(ll.Marker(
            location=(row['WGS84_N'], row['WGS84_E']),
            icon=specialless_clinic_icon,
            draggable=False
        ))
    else:
        specialful_clinic_markers.append(ll.Marker(
            location=(row['WGS84_N'], row['WGS84_E']),
            icon=specialful_clinic_icon,
            draggable=False
        ))
    
specialful_clinic_cluster = ll.MarkerCluster(markers=specialful_clinic_markers)
specialless_clinic_cluster = ll.MarkerCluster(markers=specialless_clinic_markers)

clinics_map = ll.Map(center=CENTER_OF_CZECHIA, zoom=7, min_zoom=7, max_zoom=19)
clinics_map.add_control(ll.FullScreenControl())
clinics_map.add_layer(specialful_clinic_cluster)
clinics_map.add_layer(specialless_clinic_cluster)
clinics_map

Neither the map nor the online research we did helped us to fill in the missing data. Given that department_ids and contractual_specialty_ids are essential for the task, we will have to drop the clinics that are missing them. Room for improvement lies within properly filling that data.

Ambulance Stations

ambulance_stations = pd.read_csv('./table_ZZS.csv')
ambulance_stations.head()

ambulance_stations.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 332 entries, 0 to 331
Data columns (total 10 columns):
 #   Column          Non-Null Count  Dtype  
---  ------          --------------  -----  
 0   intern_subj_id  332 non-null    int64  
 1   name            332 non-null    object 
 2   village         332 non-null    object 
 3   street          307 non-null    object 
 4   number          332 non-null    int64  
 5   zip             332 non-null    int64  
 6   WGS84_E         332 non-null    float64
 7   WGS84_N         332 non-null    float64
 8   company_id      332 non-null    int64  
 9   department_id   332 non-null    int64  
dtypes: float64(2), int64(5), object(3)
memory usage: 26.1+ KB

Some ambulance stations are missing streets, but that’s OK since we can calculate driving distances based on lat,lon.

Pre-processing

Since we were unable to fill in the missing clinics contractual_specialty_ids and department_ids, let’s drop all of the clinics that are missing them.

clinics.dropna(subset=['contractual_specialty_id'], inplace=True)
clinics.reset_index(drop=True, inplace=True)
len(clinics.index), clinics['contractual_specialty_id'].nunique()

(38081, 273)

Ambulance stations and clinic departments have the same structure and, for the purpose of the task, it would make sense to think of ambulance stations as clinics of a special type. To do this, we introduce a special type 'AST', assign it to the ambulance stations, and concatenate them with the clinics.

ambulance_stations['contractual_specialty_id'] = 'AST'
orgs = clinics.append(ambulance_stations, ignore_index=True)
len(orgs.index), orgs['contractual_specialty_id'].nunique()

(38413, 274)

We copy the coordinates to a new location column in (lat, lon) format to make working with them more convenient.

users['location'] = list(zip(users['WGS84_N'], users['WGS84_E']))
orgs['location'] = list(zip(orgs['WGS84_N'], orgs['WGS84_E']))

As we saw during the clinic analysis, most of the clinic departments share locations and can be uniquely identified by a combination of contractual_specialty_id and department_id. Let’s create a table of parent organizations with a row for each set of clinics that share their location.

orgs['specialty_department'] = list(zip(
    orgs['contractual_specialty_id'],
    orgs['department_id']
))
parent_orgs = orgs.groupby('location').agg({
    'contractual_specialty_id': set,
    'specialty_department': list,
    'location': 'first'
})
print(len(parent_orgs.index))
parent_orgs.head()

10797

This leaves us with 10,797 of the “actual” clinics that we will be computing routes to. We’ve stored a set of contractual_specialty_ids to a conctractual_specialty_id column to speed up the check for matching specialty ids.

From the start, we’ve aimed for maximum precision. The most precise way to solve the task would be to compute distances for each user individually, so we’ve implemented this straightforward solution first to identify the slowest part of the process and work from there.

It turned out that the slowest part is, not surprisingly, routing and that we will have to optimize it. Clustering users, and maybe, clinics, seemed like a good idea to explore.

The most difficult part of the task is to find a performant solution without sacrificing much of the precision.

Maps

Let’s return to the data exploration and plot parent_orgs and users.

orgs_map = ll.Map(center=CENTER_OF_CZECHIA, zoom=7, min_zoom=7, max_zoom=19)
orgs_map.add_control(ll.FullScreenControl())
orgs_map.add_layer(ll.MarkerCluster(markers=[
    ll.Marker(location=org['location']) for _, org in parent_orgs.iterrows()
]))
orgs_map

users_map = ll.Map(center=CENTER_OF_CZECHIA, zoom=7, min_zoom=7, max_zoom=19)
users_map.add_control(ll.FullScreenControl())
users_map.add_layer(ll.MarkerCluster(markers=[
    ll.Marker(location=org['location']) for _, org in users[users['village'] == 'Praha'].iterrows()
]))
users_map

It’s interesting to take a look at the Voronoi regions of different clinic types. They can provide us a way to query for candidates.

In mathematics, a Voronoi diagram is a partition of a plane into regions close to each of a given set of objects. In the simplest case, these objects are just finitely many points in the plane (called seeds, sites, or generators). For each seed there is a corresponding region, called Voronoi cells, consisting of all points of the plane closer to that seed than to any other.
— Voronoi diagram - Wikipedia

To compute the regions, we will use the geovoronoi library. Since it uses shapely polygons, we will import the shapely module too. We will also import geopandas to simplify manipulations with the locations in our pandas’ tables and convert them to shapely format.

import geopandas as gpd
import geovoronoi
import shapely.geometry as shg

Then, let’s convert our users and parent_orgs to geo-aware users_geo and orgs_geo tables.

users_geo = gpd.GeoDataFrame(
    users,
    crs='EPSG:4326',
    geometry=gpd.points_from_xy(users['WGS84_E'], users['WGS84_N'])
)

org_locs = list(zip(*parent_orgs['location']))
orgs_geo = gpd.GeoDataFrame(
    parent_orgs,
    crs='EPSG:4326',
    geometry=gpd.points_from_xy(org_locs[1], org_locs[0])
)

We will also need a helper method to convert shapely polygons to leaflet format:

def polygon_to_ll(polygon):
    if isinstance(polygon, shg.Point):
        return [(polygon.y, polygon.x)]
    if isinstance(polygon, shg.LineString):
        return [(point[1], point[0]) for point in polygon.coords]
    return [(point[1], point[0]) for point in polygon.exterior.coords]

And a simple method to plot regions for an org type:

# a polygon encompassing all of the users
all_users = users_geo.unary_union.convex_hull

def plot_voronoi_regions_for_org_by_type(org_type):
    region_polys, region_pts = geovoronoi.voronoi_regions_from_coords(
        orgs_geo[orgs_geo['contractual_specialty_id'] & {org_type}]['geometry'].tolist(),
        all_users
    )

    voronoi_map = ll.Map(center=CENTER_OF_CZECHIA, zoom=7, min_zoom=7, max_zoom=19)
    voronoi_map.add_control(ll.FullScreenControl())
    voronoi_map.add_layer(ll.Polygon(
        locations=[polygon_to_ll(poly) for poly in region_polys.values()],
        color="green",
        fill_color="green"
    ))
    return voronoi_map

We will plot Voronoi regions for one of the most common types of clinics and one of the rarest.

plot_voronoi_regions_for_org_by_type('014')

plot_voronoi_regions_for_org_by_type('504')

As you can see, some clinics have very sparse coverage; if they represent an essential specialty, there needs to be more of them.

plot_voronoi_regions_for_org_by_type('AST')

These Voronoi regions provide better insight into the clinic density and distribution.

Clusters

Next up is clustering. We decided to cluster users by distance using village information to speed up the clustering.

Utils

Vincenty’s formulae offer one of the most commonly adopted methods to compute the geodesic distance. It’s pretty precise but comparably slow in comparison to the great circle distance computation. To compute the latter, we will use the haversine formula. The haversine formula is much simpler, and therefore, much faster. Let’s see by how much.

We will use geographiclib’s implementation of Vincenty’s formula. It’s a great library of various geodesic algorithm implementations. For the haversine formula, we will use the python haversine library.

from geographiclib.geodesic import Geodesic as geo
import haversine

point_a = (50.088036196500006, 15.820675680125)
point_b = (50.036845777, 15.774167073)

%timeit geo.WGS84.Inverse(*point_a, *point_b)

76.7 µs ± 1.49 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

%timeit haversine.haversine(point_a, point_b, haversine.Unit.METERS)

1.31 µs ± 10.8 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

geo.WGS84.Inverse(*point_a, *point_b)['s12'], haversine.haversine(point_a, point_b, haversine.Unit.METERS)

(6596.260963017945, 6589.524077151891)

The difference in performance is staggering when the error margin is negligible. Given that we will only be using this distance for approximations, and both distances are very optimistic approximations of the driving distance, we will stick to the haversine distance going forward.

Before we move on, let’s import the progress bar. Things we do from this moment on will take much more time to compute, and it’s nice to monitor the progress.

from fastprogress.fastprogress import progress_bar

Below is the method that we will use to compute and plot the clusters.

def plot_user_clusters_by(key_fn, radius_m=10_000):
    clusters = {}

    for i, user in progress_bar(users_geo.iterrows(), total=len(users_geo.index)):
        key = key_fn(user)
        user_loc = user['location']
        if key not in clusters:
            clusters[key] = [[(i, user_loc)]]
        else:
            found_cluster = False
            for cluster in clusters[key]:
                first_user_location = cluster[0][1]
                distance = haversine.haversine(
                    first_user_location,
                    user['location'],
                    haversine.Unit.METERS
                )
                if distance < radius_m:
                    found_cluster = True
                    cluster.append((i, user_loc))
                    break
            if not found_cluster:
                clusters[key].append([(i, user_loc)])
                
    cluster_geometries = [
        shg.MultiPoint([(point[1][1], point[1][0]) for point in cluster])
        for cluster_group in clusters.values() for cluster in cluster_group
    ]
    print(f'found {len(cluster_geometries)} clusters')
    
    cluster_map = ll.Map(center=CENTER_OF_CZECHIA, zoom=7, min_zoom=7, max_zoom=19)
    cluster_map.add_control(ll.FullScreenControl())
    cluster_map.add_layer(ll.Polygon(
        locations=[polygon_to_ll(cluster.convex_hull) for cluster in cluster_geometries],
        color="green",
        fill_color="green"
    ))
    return cluster_map,cluster_geometries,clusters

Visualizations

First, we will compute the clusters by user villages.

key_fn = lambda user: user['village']
cluster_map, cluster_geometries, clusters = plot_user_clusters_by(key_fn)
cluster_map

found 6285 clusters

6,285 clusters is a good amount, but the clusters are quite large. Let’s compute clusters based on the user village_parts.

key_fn = lambda user: f'{user["village"]}_{user["village_part"]}'
cluster_map, cluster_geometries, clusters = plot_user_clusters_by(key_fn)
cluster_map

found 14728 clusters

15k is a good amount of clusters in terms of the number of requests we can issue to the Azure API, but the clusters are still quite large. Let’s explore 500m radius clusters.

cluser_map, cluster_geometries, clusters = plot_user_clusters_by(
    lambda user: f'{user["village"]}_{user["village_part"]}',
    radius_m=500
)
cluster_map

found 47199 clusters

These 47k clusters seem like the way to go for an efficient and precise solution.

Routing

With $200USD in MS Azure credit, we can run only 40k queries against the routing API if we want to send more than 50 of them per second. And with 47k clusters, we would definitely need to send > 40k requests.

To not sacrifice the precision and to iterate quickly, we set up a local instance of the open street routing machine. What follows is the code we used to send routing requests to it.

import requests

OSRM_URI = 'http://127.0.0.1:5000'
MAX_ATTEMPTS = 7

osrm_client = requests.Session()


def request(url, attempt=1):
    try:
        response = osrm_client.get(url)
        response.raise_for_status()
        return response
    except Exception as e:
        if attempt > MAX_ATTEMPTS:
            raise e
        return request(url, attempt + 1)

    
def get_route(lat1, lon1, lat2, lon2):
    url = f'{OSRM_URI}/route/v1/driving/{lon1},{lat1};{lon2},{lat2}?continue_straight=false'
    try:
        response = request(url)
        route = response.json()['routes'][0]
    except:
        print('[get_route] err', url)
        return float('inf'), float('inf'), ''
    return route['distance'], route['duration'] / 60, route['geometry']

Let’s make sure that it works and get some sense of the performance.

get_route(50.077033696848886, 14.440035214801874, 50.070578695594314, 14.494946221573803)

(5248, 7.403333333333333, 'evspH{icwAv@aK@{RkGsmDzIalBgBgt@b_@yCpBvL|ExKQtC')

%timeit get_route(50.077033696848886, 14.440035214801874, 50.070578695594314, 14.494946221573803)

1.86 ms ± 109 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

Solution

The solution is quite straightforward from that point on. But before we move on, let’s import numpy.

import numpy as np

We will need a fastvectorized method for the haversine distance computation.

def sort_data_haversine(points, data):
    distances = haversine.haversine_vector(
        data['location'].tolist(),
        points['location'].tolist(),
        unit=haversine.Unit.METERS,
        comb=True
    )
    return np.argsort(distances), distances

index, distances = sort_data_haversine(users[:100], parent_orgs)
index, distances

(array([[4989, 4992, 4911, ..., 3372, 3375, 3254],
        [4989, 4992, 4911, ..., 3372, 3375, 3254],
        [4989, 4992, 4911, ..., 3372, 3375, 3254],
        ...,
        [4743, 4803, 4799, ..., 3372, 3375, 3254],
        [3362, 3146, 3320, ..., 3807, 3375, 3254],
        [4827, 4803, 4799, ..., 3372, 3375, 3254]]),
 array([[133852.36820813, 133508.21170191, 133514.27117984, ...,
         138680.78292498, 134174.19127045, 140410.66063062],
        [133888.2414056 , 133543.99335324, 133550.20998843, ...,
         138626.18005503, 134154.00442703, 140355.03132468],
        [134038.57941885, 133694.36931002, 133700.5206838 , ...,
         138486.67124833, 133999.0278078 , 140216.3122906 ],
        ...,
        [132402.65539979, 132050.23378999, 132072.18742985, ...,
         138917.42961128, 137858.06810736, 140528.10016674],
        [109703.4194728 , 109348.16998337, 109376.71678102, ...,
         161563.91234588, 160672.22237329, 163126.36755815],
        [134407.99756303, 134053.95329282, 134079.57745343, ...,
         136852.37604113, 136692.76736826, 138434.94454964]]))

%timeit sort_data_haversine(users[:100], parent_orgs)

75.7 ms ± 63.3 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

75ms to compute 100*11k distances is pretty good.

ORG_TYPES = set(orgs['contractual_specialty_id'].unique())
len(ORG_TYPES)

274

Next, we will create a method to find solutions for each of the clusters. To be performant, we look for all types of clinics in a single pass over a sorted list of candidates. It’s hard to vectorize the method, so we run it in multiple threads.

Notice that the solution incorporates our knowledge that there are some clinics of the same type that share the exact location. When we encounter this, we randomly distribute users between such clinics to provide better coverage.

import concurrent.futures
import random

pool = concurrent.futures.ProcessPoolExecutor(max_workers=10)

def find_closest_orgs_for_point(point, index, distances):
    org_types_left = ORG_TYPES.copy()
    result = {}
    for i in index:
        if not org_types_left:
            break
        parent_org = parent_orgs.iloc[i]
        haversine_distance = distances[i]
        route = None
        for specialty_department in (sd for sd in parent_org['specialty_department'] if sd[0] in org_types_left):
            specialty = specialty_department[0]
            best_route = result.get(specialty, ((float('inf'),),))[0]
            if haversine_distance > best_route[0]:
                org_types_left.remove(specialty)
                continue
            if not route:
                route = get_route(*point['location'], *parent_org['location'])
            if (route[0] < best_route[0]) or (route[0] == best_route[0] and random.random() < .5):
                result[specialty] = (route, specialty_department, parent_org['location'])
    return result


def _wrapper(args):
    return find_closest_orgs_for_point(*args)


def find_closest(points, orgs: pd.DataFrame):
    index, distances = sort_data_haversine(points, parent_orgs)
    return list(pool.map(_wrapper, [
        (points.iloc[i], index[i], distances[i]) for i in range(len(index))
    ]))

users_res = find_closest(users[:1], parent_orgs)
list(users_res[0].values())[:5]

[((3473.6,
   5.298333333333333,
   '_d`oHyxjwAK{@g@a@qA|BeFvCYCSs@pCkGjBcIJsEw@cKZgGz@qAbCkAv@uBXg_@i@iFaCoFiBkJ{GaPg@aDiBq{@cCyVo@iL?qJv@gGZDAj@'),
  ('014', 20216001.0),
  (49.816801772, 14.517831226)),
 ((3473.6,
   5.298333333333333,
   '_d`oHyxjwAK{@g@a@qA|BeFvCYCSs@pCkGjBcIJsEw@cKZgGz@qAbCkAv@uBXg_@i@iFaCoFiBkJ{GaPg@aDiBq{@cCyVo@iL?qJv@gGZDAj@'),
  ('002', 20412001.0),
  (49.816801772, 14.517831226)),
 ((3476.6,
   5.341666666666667,
   '_d`oHyxjwAK{@g@a@qA|BeFvCYCSs@pCkGjBcIJsEw@cKZgGz@qAbCkAv@uBXg_@i@iFaCoFiBkJ{GaPg@aDiBq{@cCyVo@iLAaJx@wG}@R'),
  ('902', 20721001.0),
  (49.817220095, 14.517841305)),
 ((16488.6,
   28.763333333333332,
   '_d`oHyxjwAFrXf@|@rAIhBkKx@w@tDrC~GnVpOkD|EkCSzSqBpF`BzM~BtB~Kh`@|JzQvCn@nBkAwJt\\oJ|LMnAh@xC~EnE|A|DfKxY`ArG[hEuCzH~@lLkBnDwBzAuEmA}AlHwN|BoMsNoBMiAzAIdDzCfNYpGyHfI{JpC}HtFkF\\_VuDuR|@gEsCiCiH~@gM[sC_LeEcDpHm@|UXfH_@bBkExDQtCz@x@jGQxFzBfKfAbDfCvBpEEtCcDdObDvLAh\\t@xOxGtc@xLjd@l@hFXwChFaAbH}HvBqMzE{GdC_H`De@zAjHxC`AfLaGRwHnAoG}AgELmAnDb@`CpF|NdLvS?rCuIxEwEZwR{AiKp@eCh@g_@uC{WaG_V'),
  ('709', 22107293.0),
  (49.800521335, 14.42696598)),
 ((16488.6,
   28.763333333333332,
   '_d`oHyxjwAFrXf@|@rAIhBkKx@w@tDrC~GnVpOkD|EkCSzSqBpF`BzM~BtB~Kh`@|JzQvCn@nBkAwJt\\oJ|LMnAh@xC~EnE|A|DfKxY`ArG[hEuCzH~@lLkBnDwBzAuEmA}AlHwN|BoMsNoBMiAzAIdDzCfNYpGyHfI{JpC}HtFkF\\_VuDuR|@gEsCiCiH~@gM[sC_LeEcDpHm@|UXfH_@bBkExDQtCz@x@jGQxFzBfKfAbDfCvBpEEtCcDdObDvLAh\\t@xOxGtc@xLjd@l@hFXwChFaAbH}HvBqMzE{GdC_H`De@zAjHxC`AfLaGRwHnAoG}AgELmAnDb@`CpF|NdLvS?rCuIxEwEZwR{AiKp@eCh@g_@uC{WaG_V'),
  ('799', 22107493.0),
  (49.800521335, 14.42696598))]

%timeit -n1 -r1 find_closest(users[:100], parent_orgs)

33.2 s ± 0 ns per loop (mean ± std. dev. of 1 run, 1 loop each)

Our implementation can find the solution for 100 users in 33 seconds. We are ready to run it on user clusters, and before we do, we have to compute the centroids of the clusters.

clusters_flat = [cluster for cluster_group in clusters.values() for cluster in cluster_group]
clusters_restructured = [list(zip(*cluster)) for cluster in clusters_flat]
clusters_geo = gpd.GeoDataFrame(
    clusters_restructured,
    crs='EPSG:4326',
    columns=['users', 'locations'],
    geometry=cluster_geometries
)
clusters_geo.head()

cluster_centroids = clusters_geo.centroid.to_frame()
cluster_centroids['location'] = cluster_centroids.applymap(lambda centroid: (centroid.y, centroid.x))
cluster_centroids.head()

<ipython-input-56-c0091e7f21d9>:1: UserWarning: Geometry is in a geographic CRS. Results from 'centroid' are likely incorrect. Use 'GeoSeries.to_crs()' to re-project geometries to a projected CRS before this operation.

  cluster_centroids = clusters_geo.centroid.to_frame()

Once they are computed, we are ready to process all of the clusters.

BATCH_SIZE = 1000

cluster_results = []

total_iterations = np.ceil(len(cluster_centroids.index) / BATCH_SIZE)

for i in progress_bar(range(0, len(cluster_centroids.index), BATCH_SIZE), total=total_iterations):
    batch_results = find_closest(cluster_centroids[i:i+BATCH_SIZE], parent_orgs)
    cluster_results += batch_results

Running the solution on 47k clusters took 3 hours and 40 minutes, a time we are quite happy with.

The last thing left is writing the output CSV. Before we proceed, let’s remind ourselves how it should look like.

OUTPUT_COLUMNS

['policyholder_num',
 'contractual_specialty_id',
 'pzs_department_id',
 'km_to_pzs',
 'min_to_pzs',
 'zzs_department_id',
 'km_to_zzs',
 'min_to_zzs']

Because we really care about precision, we will adjust distance and driving times for each user based on the ratio between user⇔clinic and cluster centroid⇔clinic straight line distances.

import csv

M_IN_KM = 1000
SPEED = 833  # meters per hour (50kmh), used only when centroid2org distance is 0 (rare)


def adjust_route(route, user_location, centroid_location, org_location):
    centroid2org_distance = haversine.haversine(
        centroid_location,
        org_location,
        haversine.Unit.METERS
    )
    user2org_distance = haversine.haversine(
        user_location,
        org_location,
        haversine.Unit.METERS
    )
    if centroid2org_distance == 0:
        return user2org_distance, user2org_distance / SPEED, route[2]
    ratio = user2org_distance / centroid2org_distance
    return route[0] * ratio, route[1] * ratio, route[2]


with open('./output.csv', 'w') as file:
    writer = csv.writer(file)
    writer.writerow(OUTPUT_COLUMNS)
    
    for cluster_i, cluster in progress_bar(clusters_geo.iterrows(), total=len(clusters_geo.index)):
        cluster_rows = []
        ast_route, ast_org, ast_location = cluster_results[cluster_i]['AST']
        centroid_location = cluster_centroids.iloc[cluster_i]['location']
        for user_i in cluster['users']:
            user = users.iloc[user_i]
            user_loc = user['location']
            adj_ast_route = adjust_route(
                ast_route,
                user_loc,
                centroid_location,
                ast_location
            )
            for cluster_result in cluster_results[cluster_i].values():
                route, clinic, clinic_location = cluster_result
                if clinic[0] == 'AST':
                    continue
                adj_route = adjust_route(
                    route,
                    user_loc,
                    centroid_location,
                    clinic_location
                )
                cluster_rows.append([
                    user['policyholder_num'],
                    clinic[0],
                    f'{int(clinic[1]):08d}',
                    adj_route[0] / M_IN_KM,
                    adj_route[1],
                    f'{int(ast_org[1]):08d}',
                    adj_ast_route[0] / M_IN_KM,
                    adj_ast_route[1],
                ])
        writer.writerows(cluster_rows)

As you can see, we came up with an elegant, efficient, and precise solution by thoroughly analyzing the data, clustering the users, setting up the local routing machine, vectorizing and parallelizing the computations, and approximating final driving distances and times for each of the users individually.

Result visualization

Let us conclude by plotting the results for a few clusters.

import polyline

CLUSTER_COLORS = ['green', 'red', 'orange']

routes_map = ll.Map(center=CENTER_OF_CZECHIA, zoom=7, min_zoom=7, max_zoom=19)
routes_map.add_control(ll.FullScreenControl())

for i in range(len(CLUSTER_COLORS)):
    cluster_color = CLUSTER_COLORS[i]
    cluster_result = cluster_results[i * 10000]
    for j, specialty_result in enumerate(cluster_result.items()):
        _, result = specialty_result
        route = polyline.decode(result[0][2])
        routes_map.add_layer(ll.Polyline(
            locations=route,
            color=cluster_color,
            fill=False
        ))
        routes_map.add_layer(ll.Marker(
            location=route[-1],
            icon=ll.AwesomeIcon(name='hospital-o'),
            draggable=False
        ))
        if j == 0:
            routes_map.add_layer(ll.Marker(
                location=route[0],
                icon=ll.AwesomeIcon(name='users', marker_color=cluster_color),
                draggable=False
            ))
        
routes_map

This map is interactive, and results for each of the clusters are color-coded; feel free to explore it. I hope that you’ve enjoyed this article! If you have any thoughts about it, please, reach out to me on Twitter.

I want to end by saying thank you to the ML Prague Conference, SAS, and all other sponsors for the hackathon! I had a lot of fun and am already looking forward to next year! :)

You are developing a react-native app with Tensorflow JS? Then you may have tried using the tfjs-react-native backend in Android SDK Emulator only to find that it doesn’t work.

This happens because tfjs-react-native requires GLES 3 / WebGL 2. You can enable it in Android SDK Emulator via:

echo "GLESDynamicVersion = on" >> ~/.android/advancedFeatures.ini

I tested this only under Linux. Your host GPU should support at least OpenGL ES 3.2; you can check if it does via:

sudo apt install mesa-utils
glxinfo | grep 'version.*OpenGL'

People often ask me for a good place to get started with AI, machine learning, and neural nets.

course.fast.ai — this is a pretty good place. It’s modern (the current version of the course was released this August), centered around the relevant practice, teaches you by doing, going deeper only when needed, and it goes with the book that was released this August, too. Overall, it’s a pretty good introduction to the field.

There are also good courses from Andrew Ng on Coursera (his Deep Learning specialization is great), but, in my personal opinion, the fast.ai course is a better introduction to hands-on AI in the modern world.

In conclusion, I’ll recommend you to do both fast.ai and Andrew Ng’s courses because each of them offers a different perspective. The good idea is to top it all up with some practice that you can find on Kaggle (they also offer some courses on Kaggle Learn, I’m not familiar with those, though).