The goal of this project is to extract data from "3D" sudoku images, and solve them. This was used for a project for which we had a robot that had to navigate to a sudoku image, extract the data, solve the puzzle and then draw the digit that is supposed to be in the sudoku's red square.
Here's what a "3D" sudoku looks like.
The sudokus are constructed as 3 4x4 grid of numbers. Just like a regular sudoku, the same number cannot appear twice in the same row, the same column and, in this case, the same blue rectangle.
There are two main parts to the algorithm solving this problem. The first one extracts the data from the image, while the second solves the puzzle.
First of all, we have to keep in mind that the sudokus' images come from a robot that had to navigate a long way to get there. Therefore, we cannot guarantee that the image will be perfect. We have to take into account that the sudoku will be warped and that we may even get other stuff in the picture.
With that in mind, the first step of the process is to unwarp the image, in order to get a consistent image even if the robot was slightly off its course. To do so, we are lucky enough to have a green square around the puzzle. All we have to do is get the four corners of this square, and unwarp the square.
Here's an example of how that's done (in strange pinkinsh colors...)
We then have to find the corners of the "box". Doing so will enable us to define the puzzle's boundaries and the placement of the digits.
- We first isolate the puzzle's box by its color.
- Then, we use OpenCV's power to find possible corners.
- By simple geometry, we are able to find the best matches for possible corners.
Once we have the box's corners, it's a simple matter of geometry to find the possible places where digits might be. Therefore, the harder part is to actually get the right digit from the image.
In order to keep it simple, we simply trained a regular classifier using the k-nearest neighboors algorithm. To do so, we fed the classifier a series a images containing digits that were manually identified beforehand. This enabled the classifier to compare the digits in the puzzle to the digits it already knew. Because it is a simple OCR problem (same font, same size), the KNN classifier works like a charm.
Sudoku solving algorithms are abundant on the Internet. However, this particular puzzle is a little bit different from a regular sudoku. Therefore, we had to write a specific solving algorithm.
The algorithm used is largely based on a sudoku solving algorithm written by Peter Norvig. We were able to reuse that algorithm because both sukodus and "3D" sudokus are constraint satisfaction problems. Therefore, all we had to do was modify the constraints. The algorithm works in two ways.
- Constraints are propagated troughout the problem set to find a solution
- Depth first searches are tried, whenever we try a new solution.
A solution is usually found within a few milliseconds on the robot's computer (an old mac mini).
