How Mapersive Automates Feature Extraction and Asset Recognition
This article covers how Mapersive’s SaaS platform uses AI-assisted feature extraction and asset recognition on 360-degree street view imagery. We’ll cover the following topics:
- what is AI-assisted feature extraction
- how Mapersive’s data pipeline uses AI for feature extraction and asset recognition
- how geospatial imagery is georeferenced for precise positioning
- what is the role of LiDAR alongside photogrammetric imagery for Mapersive’s asset recognition data pipeline
- how does Mapersive plan to extend current AI capabilities in future releases
A Data Pipeline for Creating GIS Data from Mobile Mapping Imagery
Today, AI is applied to identify and extract relevant features in large collections of mobile mapping imagery. This approach speeds up mobile mapping workflows while guaranteeing accuracy and reducing the need for manual data processing. AI-assisted feature extraction in mobile mapping utilizes Deep Learning algorithms to analyze data from sensors and cameras on mobile mapping vehicles. The AI detects patterns and distinguishes features such as roads, buildings, vegetation, and infrastructure. These features can then be saved as spatial data, along with attributes and metadata, so that they can be used smoothly and efficiently in GIS data analysis workflows.
Connected data processing elements in series are called data pipelines, where the output of one element is the input of the next. We’ll now have a detailed look at one such data pipeline that takes 360-degree imagery and automates the feature extraction process from that data.
Breaking Down Mapersive’s Spatial Data Extraction Pipeline
Mapersive is a geospatial SaaS solution provider based in Augusta, Georgia. The company’s core offering is a spatial data extraction platform that helps companies and organizations make better-informed decisions. Mapersive aims to transform the techniques of data exploration, data extraction, and data use processes. The Mapersive platform combines advanced AI technology and algorithms for in-depth spatio-temporal data processing and extraction.
Defining 360-degree imagery
The Mapersive platform processes 360-degree imagery that is uploaded by its users. 360-degree imagery consists of panoramic images typically collected from vehicles equipped with a mobile mapping system. Such imagery provides a complete visual representation of the surroundings, which is highly useful for applications such as detailed inspections and infrastructure management.
Although the following content focuses on 360-degree imagery, the Mapersive platform is not exclusively built for uploading and processing street view image data: it can be used for any survey that would benefit from having a digital twin that is easily accessible digitally.
From raster to vector data: creating GIS features from imagery
The platform takes in 360-degree imagery that is captured with Mosaic cameras for clients with a need for data that covers large geographical areas. These are mainly public institutions that do inspection work and need GIS vector data that shows the location and state of things such as traffic lights and road signs. Such vector data can be created by extracting them from raster files, which is what images are: large collections of pixels.
Users can upload and manage 360-degree imagery captured with mobile mapping systems and experience them in an immersive way on the Mapersive platform. With imagery datasets that cover large geographical areas, extracting features is an involved, time-consuming, and repetitive task. Mapersive created a data workflow that assists the user with this task to help them develop and export spatial data (objects) in several spatial data formats such as shapefiles and GeoJSON. These can then be used for tasks such as GIS inspection workflows, among others.
How Mapersive applies assisted AI for asset recognition and feature extraction
The workflow to extract spatial objects from the imagery works as follows: the platform’s user interface enables the user to interact with the imagery, and define and select objects with their geometries directly from 360 degree imagery. Next, the platform’s algorithms compute the spatial components of that data, so that its real-world location has been defined with the spatial object. The user can then attach metadata such as object category or its condition manually.
At the recent ESRI UC 2024 Head of Global Business Development & Sales for Reality Mapping at Esri Arkadiusz Szadkowski shared the following:
“AI works better on photos (images) than LiDAR because it was trained on images”
Calculating real-world locations within imagery using AI
Mapersive’s data extraction process is AI-assisted, which combines human expertise and AI efficiency, for example, to ensure higher precision in complex or ambiguous scenarios. In the case of Mapersive, the user still has to interact with the imagery, but in a minimal fashion: AI discovers the objects inside the imagery, but the user needs to confirm the extraction operation by clicking inside the 360-degree imagery. Next, Mapersive’s assisted data extraction pipeline calculates the real-world location of the different objects based on their location inside the imagery.
Besides finding the objects inside the imagery, their real-world location and geometries are extracted. Proprietary, AI computer vision algorithms help improve the accuracy of the platform’s spatial calculations for where the real-world objects inside the imagery are located. The geometries extend to complex geometries such as line heights, distances, and surfaces.
Deploying a data scheme for creating and exporting spatial data for GIS workflows
Spatial vector data is usually stored as points, lines, and polygons. Mapersive supports 2D and 3D geometry types, such as 2D and 3D points, and 3D surfaces such as entrances, buildings’ facades, etc. Once identified, these objects can be labeled and categorized by a human operator.
Before starting the feature extraction process, the user can create a custom schema of what their data will look like. This default data categorization system is as universal as possible and can include things like urban infrastructure or utilities. Depending on the specific use case a user wants to adopt, a custom layer can be created with specific required attributes for the data extraction operations.
Creating a custom schema
There are no restrictions laid upon the user so that at any given time, any information from the imagery can be extracted, with or without added spatial context if necessary. The same goes for data fields or attributes: the user can add any information about the objects at any given time.
After the data extraction process has finished, human verification of the extracted features is required to manually verify the accuracy of the extracted features. Users will need 5-10 seconds per extracted object. Scaling that up to a city level is more manageable than sending out a team of operations to do this in the field.
Combining Cameras, Data, and Software
Mosaic’s cameras have been Mapersive’s platform of choice to acquire the 360-degree imagery stored in the Mapersive platform and processed through the feature extraction data pipeline. The following describes how their collaboration started and provided a solid foundation that expanded over time.
Benchmarking Mosaic’s mobile mapping system
Their business collaboration with Mosaic started two years ago when Mapersive reached out to multiple vendors to benchmark several mobile mapping systems. Mosaic was their first choice, as their systems provide many of the features Mapersive was looking for to both test and operate. Mapersive is working on supporting more cameras in the near future. Read more about the collaboration between Mapersive and Mosaic and how they redefine urban mapping and geospatial analysis.
Mosaic also shared raw mobile mapping imagery and some of their in-house tools with Mapersive for preprocessing raw data. This dramatically shortened data processing times from two months to a few minutes.
Capturing high-accuracy mobile mapping data
With the right equipment in place, Mapersive was able to start testing Mosaic’s solution to see if the imagery data matched their real-world location precisely. The data quality was very high, and Mapersive now recommends their camera systems to clients. In particular, Mapersive found the resulting images well-stabilized, especially in rough environments. Second, the camera system adapts well to different brightness settings and has a good dynamic range which makes it easy to identify objects far away from the camera in diverse conditions and times of day.
Mapersive’s algorithms use the GNSS data captured with the imagery to georeference the imagery in their platform. The higher the accuracy of the imagery, both in terms of image quality and GNSS locations, the higher the reliability of the 3D reconstruction results inside the platform. Again, Mapersive found the GNSS data captured with the Mosaic camera system to be adequate in terms of the margin of error to run their 3D reconstruction algorithms and get reliable results.
Why Mapersive Plans to Support LiDAR along 360-degree Imagery
Mapersive is hopeful about the future of 360-degree imagery captured with photogrammetry, as it meets all user requirements regarding accuracy and data quality. In the context of surveying, many of the company’s customers don’t need the precision of LiDAR. For city governments who need to have a general overview of inventory including lamp posts, traffic, and potholes, capturing and updating image data every few months is much easier than sifting through terabytes of LiDAR data. Also, acquiring LiDAR equipment and capturing the data is more expensive than 360 imagery.
That being said, Mapersive plans to support LiDAR uploads to its platform eventually so that users can perform the same operations on that data instead of just imagery. This will be an interesting feature for customers who have already invested in LiDAR, and enable them to exploit the resulting point cloud on Mapersive’s platform as well.
Enabling Fully Automated Feature Extraction from 360-degree Imagery
Geospatial Artificial Intelligence (GeoAI) has greatly improved the scalability of extracting spatial data from 360-degree imagery. This innovative approach leverages advanced Deep Learning and photogrammetry algorithms to automatically identify and classify features within vast collections of mobile mapping imagery while also extracting their spatial location. The benefits are manifold: increased efficiency, enhanced accuracy, and significant reduction in the need for manual data processing.
GeoAI in Automating the Extraction Process
GeoAI automates the feature extraction process by employing Deep Learning models trained on large datasets of labeled 360 degree imagery. These models can recognize and delineate various objects such as buildings, roads, trees, and other urban infrastructure with remarkable precision. By automating the extraction process, GeoAI reduces the time and effort required to convert raw imagery into ready-to-use spatial data.
Mapersive’s AI Strategy
Mapersive’s AI strategy is centered around streamlining large-scale data extraction while providing flexible tools to meet diverse user needs. One of the platform’s objectives is to offer a variety of modes of data extraction, ranging from fully automated processes to user-guided interactions, ensuring that users can choose the approach that best suits their requirements. Some of the data extraction methods that are within Mapersive’s feature road-map include:
- Fully Automated Data Extraction: Fully automated data extraction allows users to upload their 360-degree imagery and let the platform handle the rest. The GeoAI models would autonomously detect and extract relevant features, converting them into GIS-ready data without any manual intervention. This mode is ideal for large datasets where speed and efficiency are paramount.
- One-Click Data Extraction: For users who prefer a more hands-on approach, Mapersive will offer a “one-click” data extraction mode. Users will be able to simply click on areas of interest within the imagery, and the GeoAI algorithms will handle the extraction process for those specific objects by accurately scanning the dataset for object correspondences and triangulating the object’s position. This method combines ease of use with the power of GeoAI, greatly accelerating the extraction process while allowing users to manually describe whatever object characteristics required.
- AI-Assisted Quality Assurance Tools for Data Extraction: To ensure the highest accuracy, Mapersive is also working on providing AI-powered quality assurance tools. These tools will allow users to review and validate the extracted features, making corrections as needed. The GeoAI will be used to offer suggestions and highlight potential errors, streamlining the quality assurance process and enhancing the reliability of the final data.
- Providing Specialized Regional and Global GeoAI Models: Mapersive’s platform is progressively adding specialized GeoAI models tailored for both regional and global applications. These models are trained to recognize specific object types common in different geographical regions, ensuring that the extraction process is fine-tuned to local contexts, like traffic signage and local vegetation. Whether it’s identifying unique architectural styles or regional infrastructure, these models enhance the precision and relevance of the extracted data, wherever the imagery is captured.
- Orienting and Guiding Users Towards Data “Hotspots” To further enhance the user experience, Mapersive’s platform will include tools to orient and guide users towards data “hotspots” within their imagery datasets. These hotspots are areas where the GeoAI model has identified a high density of features or objects of interest. By directing users to these areas, the platform ensures that they can quickly focus on the most relevant parts of their data, easily estimate the time and effort required to extract data from their imagery, maximizing efficiency and effectiveness in their mapping projects.
FAQ
Q: What is mobile mapping?
A: Mobile mapping is used for collecting geospatial data from a moving vehicle equipped with sensors like GNSS, cameras, LiDAR, and IMUs. Mobile mapping is used for asset management, such as road signs and conditions.
Q: Does mobile mapping produce imagery, LiDAR, or both?
A: Mobile mapping can produce both imagery and LiDAR data depending on the mobile mapping system configuration. Combining cameras for high-resolution imagery and LiDAR sensors for precise 3D mapping is possible, but the resulting accuracy is often not required. For the identification of features and their conditions, capturing imagery is more than enough.
Q: Who uses mobile mapping?
A: Mobile mapping is used by urban planners, transportation agencies, utility companies, environmental researchers, and disaster management teams. These people rely on current geospatial imagery, which explains why mobile mapping data is captured regularly.
Q: How do you know the location where mobile mapping imagery is taken?
A: In mobile mapping, GNSS provides precise location data, while the IMU measures the vehicle’s orientation, velocity, and acceleration. Together, they ensure accurate geospatial positioning and mapping, allowing for precise data collection and alignment of imagery and LiDAR scans.
If you’re in the market for great hardware that will provide you with detailed images for your post-processing workflow, please contact Mosaic today.
