Meet BinoML: A Novel Machine Learning Ranking Model for Precisely Adding Building Numbers to Unlabeled Buildings

Ever wondered how the package that we order online are delivered within such a short time and so accurately? The availability of accurate addresses plays a very important role in this. Suppose the address provided by the consumer is not present on the map accurately, then the delivery person will have trouble finding the location hence the package may be delayed or can even be delivered to the wrong address. To further improve the functioning of the delivery system, a team of scientists from Amazon devised an ML method to auto-label addresses to the building using the data of the packages delivered to that particular address in the past. Even though there are free and collaborative projects for creating global geographic databases, such as OpenStreetMap (OSM) (Fig. 1), which provides building outlines with building numbers, there are still unlabeled areas (Fig. 2) in the United States.  In this paper, the model is tested for the US but can be easily applied to other countries also after some fine-tuning to new regions.

There are some pre-existing approaches that address this issue; the best of the past approach is a scalable heuristic algorithm that matches an address to a building outline and uses the building number from the address text to label the corresponding building. The heuristic method for each address takes its latitude and longitude representation from the geocode1 file. Calculate a confidence score for all candidate buildings in a 30m radius based on geocode distance and Choose the nearest building. Then remove the matches that label a building with ambiguous numbers or have confidence less than 0.95.

The DP(delivery point) model uses ranking to choose the best delivery scan point as the geocode of an address (Fig. 4). However, the drivers may not scan the packages only at their dropping locations to confirm delivery. That’s why we need to find the best delivery scan point. There are still scenarios in which the best delivery scan point lies between two buildings (Fig. 5), hence confusing the model to which building it should assign the address.  

In this paper, the researchers proposed a rank-based approach for assigning addresses to the correct building, but in spite of scan points for an address, they created a set of building-related features and ranked candidate buildings for each address. This will avoid assigning the same address to multiple buildings but still allow a building to have multiple addresses. 

Since it is an ML model, we will need data to train it; what about that?

The data consists of 18 months of package scan data of a delivery region, a road segment map, and OSM’s building outlines of the delivery region. There is some pre-processing done before feeding the data to the model. In preprocessing, they removed ambiguous addresses using a balking classifier from the DP Model. They also normalized the addresses with a structure like “APT Number!Building Number!Street Name!City!County!State!Country”.  Common Abbreviations and unnecessary spaces are also removed. In segmented maps and OSM’s building that doesn’t deserve an address label is removed (like a garage, sheds, etc.) using size as a parameter(<30 sq. meters). After all this, a DP for each address is obtained, and buildings around that point up to a distance are taken as candidates for that address. In some cases, the buildings along a road segment follow a sequential order, so this information is captured by assigning a positional order to the buildings.

Now from this preprocessed data, feature vectors are created. Even though there are more than 25 features created, the major 10  features include the following:

1. KDE (2d kernel density estimate) distance: Minimum distance between a building and max KDE score point. 
2. Geocode distance: Minimum distance between a building and the latest DP point of an address.
3. Inbetween: If the address text has a building number next or previous to the target building.
4. Inside building scan share: Ratio of scans inside this building to scans inside any building.
5. Soft vote share: Each scan of an address casts a partial vote to a candidate building, which has a weightage inversely proportional to the distance between the scan point and the building.
6. Average scan distance to a building
7. Relative building area: Z-score value of a building outline’s area among the area of all candidate buildings for an address.
8. Name difference: The difference between the building number in the address text and the building’s labeled number.
9. Position means The absolute mean of non-NAN differences between an address’s building number and a building’s neighbors’ labeled numbers.

There are some background features for an address also, which include information such as maximum soft vote share, number of candidate buildings, the ratio of scans within 5m and 20m of the building, etc. After forming all possible pairs from candidate buildings of an address, a feature vector (v-u,c) is created, where v and u refer to features of right and left buildings, respectively, and c is the common background features of an address.

To train the model, ground truth data from Nashville TN (medium building density), Chicago IL (high building density), and Fort Myers FL (mixed building density) is taken. Then feature vectors are created as described above and the ground truth dataset is split into 75% (60000 addresses) train data and 25% (20000 addresses)test data. Randomly place the correct building in right or left in pairs to create binary target, which decides whether left building is better than right building for an address or not. A Random forest binary classifier is trained 5 fold cross-validation and best model is selected based on accuracy and ROC AUC score on the test data.

For inference for an address, they pick a building which is better than all other candidates.

Auditors are used for evaluating the model. Auditors randomly selected 1000 samples from the BinoML predictions and classify each building address pair as correct match or incorrect match. A model threshold of 0.8 is used so that the precision in automatic labelling of buildings is >=99%. More results can be seen in the below Tables. On analysing the incorrect matches, it was noticed that most of the matches are due to addresses being assigned to non-residential buildings like garage, sheds etc. 

In conclusion, this model has the potential to highly contribute to optimizing delivery service and reduce the number of delivered but not received events due to more labelled buildings and more information available to drivers. It will also reduce the cost of purchasing this information from a third-party vendor. 

BinoML: A Supervised ranking method for labeling buildings

Have you ever wondered how the packages that we order online are delivered within such a short time and so accurately? The availability of an accurate address plays a very important role in this. Suppose the address provided by the consumer is not present on the map accurately, then the delivery person will need help finding the location; hence the package may be delayed or can even be delivered to the wrong address. To further improve the functioning of the delivery system, a team of scientists from Amazon devised an ML method to auto-label addresses to the building using the data of the packages delivered to that particular address in the past. Even though there are free and collaborative projects for creating global geographic databases, such as OpenStreetMap (OSM) (Fig. 1), which provides building outlines with building numbers, there are still unlabeled areas (Fig. 2) in the United States. In this paper, the model is tested for the US but can be easily applied to other countries after some fine-tuning to new regions.

Some pre-existing approaches address this issue; the best of the past methods is a scalable heuristic algorithm that matches an address to a building outline and uses the building number from the address text to label the corresponding building. The heuristic method for each address takes its latitude and longitude representation from the geocode1 file. Calculate a confidence score for all candidate buildings in a 30m radius based on geocode distance, and choose the nearest building. Then remove the matches that label a building with ambiguous numbers or have confidence less than 0.95.

The DP(delivery point) model uses ranking to choose the best delivery scan point as the geocode of an address (Fig. 4). However, the drivers may not scan the packages only at their dropping locations to confirm delivery. That’s why we need to find the best delivery scan point. There are still scenarios where the best delivery scan point lies between two buildings (Fig. 5), confusing the model to which building it should assign the address to.  

In this paper, the researchers proposed a rank-based approach for assigning an address to the correct building. Still, despite scan points for an address, they created a set of building-related features and ranked candidate buildings for each address. This method prevents the same address from being assigned to multiple buildings while allowing a building to have multiple addresses. 

Since it is an ML model, we will need data to train it. What about that?

The data consists of 18 months of package scan data of a delivery region, road segment maps, and OSM’s building outlines of the delivery region. There is some preprocessing done before feeding the data to the model. In preprocessing, they removed ambiguous addresses using a balking classifier from the DP Model. They also normalized the addresses with a structure like “APT Number!Building Number!Street Name!City!County!State!Country”. Common Abbreviations and unnecessary spaces are also removed. In segmented maps and OSM’s buildings that don’t deserve an address label are removed (like garages, sheds, etc.) using size as a parameter(<30 sq. meters). A DP is made for each address, and buildings within a certain distance around that point are chosen candidates for that address. In some cases, the buildings along a stretch of road are in a certain order (Fig. 8). This information is stored by giving each building a positional order.

Now from this preprocessed data, feature vectors are created. Even though there are more than 25 features built, some of them are as follows:

1. KDE (2d kernel density estimate) distance: Minimum distance between a building and max KDE score point. 
2. Geocode distance: Minimum distance between a building and the latest DP point of an address.
3. Inbetween: If the address text has a building number next or previous to the target building.
4. Inside building scan share: Ratio of scans inside this building to scans inside any building.
5. Soft vote share: Each scan of an address casts a partial vote to a candidate building, which has a weightage inversely proportional to the distance between the scan point and the building.
6. Average scan distance to a building
7. Relative building area: Z-score value of a building outline’s area among the area of all candidate buildings for an address.
8. Name difference: The difference between the building number in the address text and the building’s labeled number.
9. Position mean: The absolute mean of non-NAN differences between an address’s building number and a building’s neighbors’ labeled numbers.

There are some background features for an address also, which include information such as maximum soft vote share, number of candidate buildings, the ratio of scans within 5m and 20m of the building, etc. After forming all possible pairs from an address’s candidate buildings, a feature vector (v-u, c) is created, where v and u refer to features of the right and left buildings, respectively. c is the address’s common background features.

To train the model, ground truth data from Nashville TN (medium building density), Chicago IL (high building density), and Fort Myers FL (mixed building density) is taken. Then, as previously described, feature vectors are generated, and the ground truth dataset is divided into 75% training data (60000 addresses) and 25% test data (20000 addresses). Randomly place the correct building on the right or left in pairs to create a binary target, deciding whether the left building is better than the right building for an address. A Random forest binary classifier is trained 5-fold cross-validation, and the best model is selected based on accuracy and ROC AUC score on the test data.

For inference for an address, we pick a building that is better than all other candidates.

Auditors are used for evaluating the model. Auditors picked 1000 samples randomly from the BinoML predictions and decided whether each pair of building addresses was a good match or not (Fig. 13). A model threshold of 0.8 is used so that the precision in the automatic labeling of buildings is >=99%. On analyzing the incorrect matches, it is noticed that most matches are due to addresses assigned to non-residential buildings like garages, sheds, etc. More results can be seen in below Tables. 

In conclusion, this model has the potential to highly contribute to optimizing delivery service and reduce the number of delivered but not received events due to more labeled buildings and more information available to drivers. It will also reduce the cost of purchasing this information from a third-party vendor. 

Check out the Paper. All Credit For This Research Goes To Researchers on This Project. Also, don’t forget to join our Reddit page and discord channel, where we share the latest AI research news, cool AI projects, and more.