A10-3588 License Plate Recognition: Model Conversion and NPU Execution

willow1371
Jun 3
4 min read

Introduction

This document provides a detailed guide on converting the License Plate Recognition model and performing inference on the A10 - 3588 NPU.

License Plate Recognition consists of four main parts.

Firstly, there is license plate detection. We utilize the YOLOv8n - pose algorithm to detect license plates.
Secondly, license plate localization is carried out. This step precisely determines the position of the detected license plate within the image.
Thirdly, character recognition is performed. The Densenet Ctc model is employed to recognize the characters on the license plate.
Finally, the VGG16 model is used to identify the color of the license plate.

Convert Model

Lpd(YOLOV8n-pose)

git clone https://github.com/jahongir7174/YOLOv8-pose.git

Before training, please modify as follows.

For YOLOv8-pose/nets/nn.py.

         self.fpn = DarkFPN(width, depth)

         img_dummy = torch.zeros(1, 3, 256, 256)

-        self.head = Head(num_classes, (17, 3), (width[3], width[4], width[5]))

+        self.head = Head(num_classes, (4, 3), (width[3], width[4], width[5]))

         self.head.stride = torch.tensor([256 / x.shape[-2] for x in self.forwar

 d(img_dummy)[0]])

         self.stride = self.head.stride

         self.head.initialize_biases()

For YOLOv8-pose/utils/args.yaml.

flip_lr: 0.5000               # image flip left-right (probability)

mosaic: 1.00000               # image mosaic (probability)

mix_up: 0.00000               # image mix-up (probability)
-kpt_shape: [ 17, 3 ]         # number of key-points, number of dims (2 for x,y or 3 for x,y,visible)

+kpt_shape: [ 4, 3 ]          # number of key-points, number of dims (2 for x,y or 3 for x,y,visible)

-flip_index: [ 0, 2, 1, 4, 3, 6, 5, 8, 7, 10, 9, 12, 11, 14, 13, 16, 15 ]

+flip_index: [ 0, 1, 2, 3 ]

 # classes

 names:
-  0: person

+  0: license_plate

For YOLOv8-pose/utils/util.py.

-KPT_SIGMA = numpy.array([.26, .25, .25,
-                         .35, .35, .79,
-                         .79, .72, .72,
-                         .62, .62, 1.07,
-                         1.07, .87, .87,
-                         .89, .89]) / 10.0

+KPT_SIGMA = numpy.array([0.25, 0.25, 0.25, 0.25])

After training, modify YOLOv8-pose/nets/nn.py as follows.

def forward(self, x):

        x_det = self.detect_box(x)

        x_kpt = self.detect_kpt(x)

        if self.training:

            return x_det, x_kpt -       return torch.cat([x_det, self.decode_kpt(x_kpt)], dim=1)

+       output = []

+       print(x_det.shape)

+       print(x_kpt[0].shape)

+       output.append((torch.cat([x_det[:, :, :6400], x_kpt[0]], dim=1)+).view(1, 1, -1, 80, 80).permute(0, 1, 3, 4, 2))

+       output.append((torch.cat([x_det[:, :, 6400:8000], x_kpt[1]], di+m=1)).view(1, 1, -1, 40, 40).permute(0, 1, 3, 4, 2))

+       output.append((torch.cat([x_det[:, :, 8000:], x_kpt[2]], dim=1)+).view(1, 1, -1, 20, 20).permute(0, 1, 3, 4, 2))

+       return output

    def detect_box(self, x):

        x_det = []

        shape = x[0].shape[0]

        for i in range(self.nl):

            x_det.append(torch.cat((self.box[i](x[i]), self.cls[i](x[i])), dim=1

))

        if self.training:

            return x_det

        self.anchors, self.strides = (x.transpose(0, 1) for x in make_anchors(x_det, self.stride, 0.5))

        x_cat = torch.cat([i.view(shape, self.no, -1) for i in x_det], dim=2)

+       return x_cat

        box, cls = x_cat.split((self.ch * 4, self.nc), 1)

        a, b = self.dfl(box).chunk(2, 1)

        a = self.anchors.unsqueeze(0) - a

        b = self.anchors.unsqueeze(0) + b

        box = torch.cat(((a + b) / 2, b - a), dim=1)

        return torch.cat((box * self.strides, cls.sigmoid()), dim=1)

def detect_kpt(self, x):

        x_kpt = []

        shape = x[0].shape[0]

        for i in range(self.nl):

            kpt = self.kpt[i](x[i])

            kpt = kpt.view(shape, self.num_kpt, -1)

            x_kpt.append(kpt)
-       return torch.cat(x_kpt, dim=-1)

+       return x_kpt

Create export.py and write in follow code.

from nets import nn

import torch

import yaml

from utils.util import load_weight

model_path = "weights/epoch_397.pt"

save_path = "yolov8_pose.onnx"

with open('utils/args.yaml', errors='ignore') as f:

    params = yaml.safe_load(f)

model = nn.yolo_v8_n(len(params['names']))

model = load_weight(model_path, model)

# print(model)

img = torch.zeros(1, 3, 640, 640)

torch.onnx.export(model, img, save_path, verbose=False, opset_version=12, input_names=['images'])

Run export.py to convert model from pt to onnx.

python export.py

Download the convert tool.

git clone https://github.com/rockchip-linux/rknn-toolkit2.git

Create a docker.

docker pull yanwyb/npu:v1

$ docker run -it --name {name} -v $(pwd):/home/khadas/npu \

                               -v /etc/localtime:/etc/localtime:ro \

                               -v /etc/timezone:/etc/timezone:ro \ 
                               -v $HOME/.ccache:/home/khadas/.ccache --privilege

d \

                               --device=/dev/loop-control:/dev/loop-control \

                               --device=/dev/loop0:/dev/loop0 --cap-add SYS_ADMI

N yanwyb/npu:v1

Install dependencies.

cd rknn-toolkit2

sudo apt-get install python3 python3-dev python3-pip

sudo apt-get install libxslt1-dev zlib1g-dev libglib2.0 libsm6 libgl1-mesa-glx libprotobuf-dev gcc cmake

pip3 install -r doc/requirements_cp38-*.txt

pip3 install packages/rknn_toolkit2-*-cp38-cp38-linux_x86_64.whl

Modify test.py as follows.

# Create RKNN object

    rknn = RKNN(verbose=True)

    # pre-process config

    print('--> Config model')

    rknn.config(mean_values=[[0, 0, 0]], std_values=[[255, 255, 255]], target_platform='rk3588')

    print('done')

    # Load ONNX model

    print('--> Loading model')

    ret = rknn.load_onnx(model=ONNX_MODEL)

    if ret != 0:

        print('Load model failed!')

        exit(ret)

    print('done')

    # Build model

    print('--> Building model')

    ret = rknn.build(do_quantization=False, dataset=DATASET)

    if ret != 0:

        print('Build model failed!')

        exit(ret)

    print('done')

    # Export RKNN model

    print('--> Export rknn model')

    ret = rknn.export_rknn(RKNN_MODEL)

    if ret != 0:

        print('Export rknn model failed!')

        exit(ret)

    print('done')

Run test.py.

python3 test.py

Lpr(Densenet Ctc)

git clone https://github.com/YCG09/chinese_ocr.git

Before converting the model, we suggest that you convert Keras model into ONNX model first. The converting code is as follows.

import onnx

from keras.models import *

import keras

import keras2onnx

from train import get_model

import densenet

basemodel, model = get_model(32, 88)

basemodel.load_weights("models/weights_densenet-32-0.40.h5")

onnx_model = keras2onnx.convert_keras(basemodel, basemodel.name, target_opset=12)

onnx_model.graph.input[0].type.tensor_type.shape.dim[0].dim_value = int(1)

onnx_model.graph.input[0].type.tensor_type.shape.dim[1].dim_value = int(1)

onnx_model.graph.input[0].type.tensor_type.shape.dim[2].dim_value = int(32)

onnx_model.graph.input[0].type.tensor_type.shape.dim[3].dim_value = int(280)

onnx_model.graph.output[0].type.tensor_type.shape.dim[0].dim_value = int(1)

onnx_model.graph.node.remove(onnx_model.graph.node[0])

onnx_model.graph.node[0].input[0] = "the_input"

onnx.save_model(onnx_model, "./densenet_ctc.onnx")

Put densenet_ctc model and quantized picture in densenet_ctc folder.

cd densenet_ctc

cp -r {path}/densenet_ctc.onnx ./

Modify the parameters of test.py, including mean values, normalized values, the path to the Densenet-CTC model, and quantized images.

When you have run test.py successfully, you will find an RKNN model in your save path.

Lpc(VGG16)

git clone https://github.com/Daipuwei/Mini-VGG-CIFAR10

The code uses Keras. So, we should convert the model from h5 to pb. Here is the tool we use.

git clone https://github.com/amir-abdi/keras_to_tensorflow.git

Download the Edge2 conversion tool.

Modify the parameters of test.py, including mean values, normalized values, the VGG-16 model path, and quantized images.

When you have run test.py successfully, you will find an RKNN model in your save path.

Run NPU

Connect A10-3588 and pull npu code in A10-3588.

git clone https://github.com/khadas/a10-3588-npu

cd a10-3588-npu/C++

Install dependencies. If you use Yocto, you do not need to do this step.

sudo apt update

sudo apt install cmake libopencv-dev

The demo is to recognize Chinese license plates. So you should install a Chinese font library first.

sudo mkdir -p /usr/share/fonts/my_fonts

sudo cp -r license_plate_recognition/data/simfang.ttf /usr/share/fonts/my_fonts

cd /usr/share/fonts/my_fonts

sudo apt install mkfontscale

mkfontscale

Then reboot Edge2. After reboot, check installation success.

fc-list :lang=zh

Copy lpd, lpr and lpc models to license_plate_recognition/data/model.

bash build.sh

cd install/license_plate_recognition

./license_plate_recognition data/model/lpd.rknn

data/model/lpr.rknn data/model/lpc.rknn ./data/img/粤 AA63D8.png

The camera input demo is the license_plate_recognition_cap.

bash build.sh

cd install/license_plate_recognition_cap

./license_plate_recognition_cap data/model/lpd.rknn

data/model/lpr.rknn data/model/lpc.rknn 33

A10-3588 License Plate Recognition: Model Conversion and NPU Execution

Introduction

Convert Model

Run NPU

Recent Posts

Comments

SUBSCRIBE FOR UPDATES