You are here: Start » PyAVL » Interoperability with Other Libraries

Interoperability with Other Libraries

PyAVL objects can be exchanged with the most common Python libraries for numerical computing and image processing: NumPy, Pillow (PIL) and OpenCV (cv2). The conversions use standard Python mechanisms, such as the buffer protocol, bytes objects and nested lists. No additional converter modules are required.

The libraries themselves are not part of Aurora Vision Library and have to be installed separately, for example with pip install numpy pillow opencv-python.

Summary

From	To	How	Pixel data
`AVL.Image`	NumPy array	`np.frombuffer(image, ...)`	Shared with the image
NumPy array	`AVL.Image`	`AVL.Image(width, height, type, depth, array.tobytes())`	Copied
`AVL.Image`	Pillow image	`PIL.Image.frombuffer(...)`	Independent of the image
Pillow image	`AVL.Image`	`AVL.Image(..., pil_image.tobytes())`	Copied
`AVL.Image`	OpenCV image	NumPy array and `cv2.cvtColor`	Copied
OpenCV image	`AVL.Image`	`cv2.cvtColor` and `AVL.Image(...)`	Copied
`AVL.Matrix`	NumPy array	`np.array(matrix.tolist())`	Copied
NumPy array	`AVL.Matrix`	`AVL.Matrix(array.tolist())`	Copied
`AVL.ByteBuffer`	`bytes`, `bytearray`, `memoryview`, NumPy array	`bytes(buffer)`, `np.frombuffer(buffer, ...)`	`bytes` and `bytearray` are copies; `memoryview` and NumPy share the data
`bytes`, `bytearray`	`AVL.ByteBuffer`	`AVL.ByteBuffer(data)`	Copied

Image Memory Layout

The conversions of images depend on how an AVL.Image stores its pixels:

The pixels are stored row by row. Every row occupies image.Pitch bytes, which can be larger than image.Width * image.PixelSize because the rows are padded for alignment.
The components of a pixel (channels) are stored next to each other. The number of channels is given by image.Depth and ranges from 1 to 4.
For color images the channels are in the order red, green, blue, the same as the components of AVL.Pixel(r, g, b). This is the order used by Pillow, but not by OpenCV, which uses blue, green, red.
The type of the channel values is image.Type. It corresponds to a NumPy data type as follows:

`AVL.PlainType`	NumPy data type
`Int8`	`np.int8`
`UInt8`	`np.uint8`
`Int16`	`np.int16`
`UInt16`	`np.uint16`
`Int32`	`np.int32`
`Real`	`np.float32`

NumPy

AVL.Image supports the Python buffer protocol, so NumPy can read and write the pixels of an image without copying them. The helper functions below take the row padding and the channel type into account. The resulting array has the shape (height, width, depth).

import numpy as np
import AVL

NUMPY_TYPES = {
    AVL.PlainType.Int8: np.int8,
    AVL.PlainType.UInt8: np.uint8,
    AVL.PlainType.Int16: np.int16,
    AVL.PlainType.UInt16: np.uint16,
    AVL.PlainType.Int32: np.int32,
    AVL.PlainType.Real: np.float32,
}

def image_to_numpy(image):
    """Returns an array that shares the memory of the image."""
    dtype = np.dtype(NUMPY_TYPES[image.Type])
    rows = np.frombuffer(image, dtype=np.uint8).reshape(image.Height, image.Pitch)
    pixels = rows[:, :image.Width * image.PixelSize].view(dtype)
    return pixels.reshape(image.Height, image.Width, image.Depth)

def numpy_to_image(array):
    """Returns a new image with a copy of the array data."""
    array = np.ascontiguousarray(array)
    if array.ndim == 2:
        array = array[:, :, np.newaxis]
    height, width, depth = array.shape
    plain_type = next(t for t, n in NUMPY_TYPES.items() if np.dtype(n) == array.dtype)
    return AVL.Image(width, height, plain_type, depth, array.tobytes())

A two-dimensional array is converted to a single-channel image. An array with a data type that is not listed in the table above has to be converted first, for example with array.astype(np.float32).

image = AVL.Image()
AVL.TestImage(inImageId=AVL.TestImageId.Lena, outRgbImage=image)

pixels = image_to_numpy(image)          # shape (height, width, 3), dtype uint8
print(pixels.mean(axis=(0, 1)))         # mean value of each channel

pixels[:50, :50] = (255, 0, 0)          # modifies the image as well
del pixels                              # release the buffer, see below

gradient = np.tile(np.arange(256, dtype=np.uint8), (100, 1))
gray = numpy_to_image(gradient)         # a 256 x 100 single-channel image

Shared Memory

An array returned by image_to_numpy refers directly to the memory of the image. Changing the array changes the image. While such an array (or any view of it) exists, the image is locked: it cannot be used as the output of a function or be initialized again. Doing so raises BufferError. Delete the array when it is no longer needed, or work on a detached copy created with image_to_numpy(image).copy().

view = image_to_numpy(image)
try:
    AVL.MultiplyImage(image, 2, image)   # raises BufferError, the image is locked
except BufferError as e:
    print(e)

del view
AVL.MultiplyImage(image, 2, image)       # works

Pillow

Pillow works with 8-bit images in the modes L (one channel), RGB (three channels) and RGBA (four channels). These correspond to AVL.Image objects of the type UInt8 and the depth 1, 3 and 4. Pillow reads the pixels using the pitch of the image, so no row padding has to be handled. In the opposite direction, images in other modes are converted to RGB first.

from PIL import Image as PilImage
import AVL

def image_to_pil(image):
    mode = {1: "L", 3: "RGB", 4: "RGBA"}[image.Depth]
    return PilImage.frombuffer(mode, (image.Width, image.Height), image, "raw", mode, image.Pitch, 1)

def pil_to_image(pil_image):
    depths = {"L": 1, "RGB": 3, "RGBA": 4}
    if pil_image.mode not in depths:
        pil_image = pil_image.convert("RGB")
    return AVL.Image(pil_image.width, pil_image.height, AVL.PlainType.UInt8,
                     depths[pil_image.mode], pil_image.tobytes())

# AVL image to Pillow
image = AVL.Image()
AVL.TestImage(inImageId=AVL.TestImageId.Lena, outRgbImage=image)
pil_image = image_to_pil(image)
pil_image.save("lena.jpg")

# Pillow to AVL image
loaded = pil_to_image(PilImage.open("photo.png"))
AVL.SaveImage(loaded, "photo_avl.png")

OpenCV

OpenCV represents images as NumPy arrays. A monochrome image (one channel) is a two-dimensional array of the shape (height, width), and a color image is a three-dimensional array with the channels in the order blue, green, red (and alpha last). Converting an image between Aurora Vision Library and OpenCV is therefore a conversion to a NumPy array combined with a swap of the red and blue channels for color images.

The channels are swapped with cv2.cvtColor. This function accepts only the channel types UInt8, UInt16 and Real (uint8, uint16 and float32 in NumPy), so the functions below check the type of a color image first and raise TypeError for any other type. Monochrome images of all types can be converted. The results are always new arrays, so the image is not locked afterwards.

import cv2
import numpy as np
import AVL

# Types of color images supported by cv2.cvtColor
CV_COLOR_IMAGE_TYPES = (AVL.PlainType.UInt8, AVL.PlainType.UInt16, AVL.PlainType.Real)
CV_COLOR_DTYPES = (np.uint8, np.uint16, np.float32)

def image_to_cv2(image):
    if image.Depth in (3, 4) and image.Type not in CV_COLOR_IMAGE_TYPES:
        raise TypeError(f"cv2.cvtColor does not support color images of the type {image.Type}")

    array = image_to_numpy(image)               # see the NumPy section
    if image.Depth == 1:
        return array[:, :, 0].copy()            # monochrome image: 2D array
    if image.Depth == 3:
        return cv2.cvtColor(array, cv2.COLOR_RGB2BGR)
    if image.Depth == 4:
        return cv2.cvtColor(array, cv2.COLOR_RGBA2BGRA)
    return array.copy()                         # other channel counts are not reordered

def cv2_to_image(array):
    if array.ndim == 3 and array.shape[2] in (3, 4):
        if array.dtype not in CV_COLOR_DTYPES:
            raise TypeError(f"cv2.cvtColor does not support color images of the type {array.dtype}")
        code = cv2.COLOR_BGR2RGB if array.shape[2] == 3 else cv2.COLOR_BGRA2RGBA
        array = cv2.cvtColor(array, code)
    return numpy_to_image(array)                # a 2D array becomes a monochrome image

# Process an OpenCV image with AVL and return to OpenCV
bgr = cv2.imread("photo.png")
avl_image = cv2_to_image(bgr)

smoothed = AVL.Image()
AVL.SmoothImage_Gauss(avl_image, smoothed, inStdDevX=3.5)

cv2.imwrite("smoothed.png", image_to_cv2(smoothed))

Matrices and NumPy

AVL.Matrix stores 32-bit floating-point values and does not export a buffer. A matrix is converted to NumPy through the nested list returned by tolist(), and a NumPy array is converted to a matrix through the nested list returned by the array's own tolist(). Both conversions copy the data.

The basic matrix arithmetic, including the matrix product @, the transposition and the inversion, is available directly in PyAVL (see Working with Matrices), so a conversion is needed only for operations that PyAVL does not provide, such as decompositions or the solution of least-squares problems. The example computes the pseudo-inverse of a matrix with NumPy and continues with the PyAVL operators:

import numpy as np
import AVL

# A matrix with 3 rows and 2 columns
a = AVL.Matrix([[1, 0], [1, 1], [1, 2]])

# Matrix to NumPy, and the pseudo-inverse computed by NumPy
a_numpy = np.array(a.tolist(), dtype=np.float32)
pinv_numpy = np.linalg.pinv(a_numpy)

# NumPy to Matrix (a matrix with 2 rows and 3 columns)
pinv = AVL.Matrix(pinv_numpy.tolist())

# The matrix product of PyAVL: the result is close to the 2x2 identity matrix
print((pinv @ a).tolist())

A NumPy array cannot be passed directly to AVL.Matrix, because the elements of a float32 array are NumPy scalars, which are not accepted as numbers. Always pass array.tolist(). Also, np.array(matrix) does not create a numerical array, because a matrix does not export a buffer: NumPy creates an array that holds the matrix object itself.

The same approach converts geometry data, for example a list of points:

points = [AVL.Point2D(1.5, 2.5), AVL.Point2D(3, 4)]

array = np.array([(p.X, p.Y) for p in points], dtype=np.float32)     # points to NumPy
points = [AVL.Point2D(x, y) for x, y in array.tolist()]              # NumPy to points

Transformation matrices of 3D points can be applied either with the @ operator of PyAVL (see Working with Matrices) or with NumPy; both apply the same transformation.

Byte Buffers

AVL.ByteBuffer holds a block of bytes, for example the content of a file or of a network message. It supports the buffer protocol, so it can be converted to the standard Python types and to NumPy arrays, and it can be created from bytes and bytearray objects.

import numpy as np
import AVL

# bytes to ByteBuffer
buffer = AVL.ByteBuffer(b"\x01\x02\x03\x04")

# ByteBuffer to bytes, bytearray, memoryview and NumPy
data = bytes(buffer)                           # a copy of the content
editable = bytearray(buffer)                   # a modifiable copy
view = memoryview(buffer)                      # shares the content with the buffer
values = np.frombuffer(buffer, dtype=np.uint8) # shares the content with the buffer

# NumPy to ByteBuffer
samples = np.array([1, 2, 3], dtype=np.uint16)
buffer = AVL.ByteBuffer(samples.tobytes())     # 6 bytes

# A file as ByteBuffer
with open("data.bin", "rb") as f:
    file_buffer = AVL.ByteBuffer(f.read())

A NumPy array cannot be passed directly to AVL.ByteBuffer; convert it with array.tobytes() first. The number of bytes of a buffer is available with len(memoryview(buffer)) or buffer.GetData().nbytes.

As in the case of images, a memoryview or a NumPy array that shares the content locks the buffer. Initializing the buffer again while such a view exists raises BufferError. Release the views first, or use the copies returned by bytes(buffer) and bytearray(buffer).