Displaying Large Images with ImageBox at High Frame Rates

[dr]

Hi,

I have been using your terrific control with excellent results. However my application now has a new requirement to display live images from a camera (4K x 3K x 16-bits) at > 15FPS.

My current display algorithm scales the input image from 16-bits to 256 bits (stored in Emgu/OpenCV Mat classes), then creates an RGB bitmap and sets it to the "Image" property of the control.  The best performance I can achieve is < 5 FPS (see simplified code below). 

Have you used your control with OpenGL in C# to display images, and if so, do you have any example code?  Or do I need to start from scratch with something like OpenTK to create my own control and handle zooming, etc. on my own?  I did use OpenGL in C++ about 10 years ago, so I could bite the bullet and start down this path.  Or is there a better method?

void GenerateBitmap()
{   ...
    unsafe
    {
        ...
        //Both m_RgbData and tempImage8BitBuffer are byte arrays
        fixed (byte* pRGB = &m_RgbData[0])
        {
            fixed (byte* pImageData = &tempImage8BitBuffer[0])
            {
                byte value;
                byte* pDst = pRGB;
                byte* pSource = pImageData;

                for (int y = 0; y < rows; y++)
                {
                    for (int x = 0; x < cols; x++)
                    {
                        value = *pSource++;                       
                        *pDst++ = value; // R
                        *pDst++ = value; // G
                        *pDst++ = value; // B

                    }//for (int x = 0; x < cols; x++)

                }//for (int y = 0; y < rows; y++)

            }//fixed (byte* pImageData = &tempImage8BitBuffer[0])

        }//fixed (byte* pRGB = &m_RgbData[0])

    }//unsafe
    ...
    m_Bitmap = new Bitmap((int)nCurrentWidth, (int)nCurrentHeight,  System.Drawing.Imaging.PixelFormat.Format24bppRgb);
    BitmapData bmd = m_Bitmap.LockBits(new Rectangle(0, 0, (int)nCurrentWidth, (int)nCurrentHeight),
        System.Drawing.Imaging.ImageLockMode.ReadOnly, m_Bitmap.PixelFormat);   
    ...
    unsafe
    {
        byte* pDst = (byte*)bmd.Scan0;
        for (int i = 0; i < nCurrentHeight; i++)
        {
            Marshal.Copy(m_RgbData, (int)(i * nCurrentWidth * 3), (IntPtr)pDst, (int)(nCurrentWidth * 3));
            pDst += bmd.Stride;
        }
    }
    m_Bitmap.UnlockBits(bmd);

    imageBox.Image = m_Bitmap;   
}


Thanks!

[Richard Moss]

Hello,

Welcome to the forums, glad you found the control useful. And thanks for the interesting question!

I've never considered desktop applications in terms of FPS before; it's not something that has ever been a factor for the typical application I work with. With that said, I would have thought repainting the control 15 times a second should be more than doable, even at such huge resolutions.

Have you tried profiling your application to find out where the bottlenecks are? As far as I know, the code you're doing with unsafe pointers is the fastest way to manipulate bitmaps in C#, but I would still wonder if it's fast enough. On the other hand, I discovered that ImageBox has a performance hit if you continuously repaint it with a unchanged zoomed image, so there's potential for improvement there too. It could even be something else such as garbage collection. I'm not sure what to suggest without more information, I definitely think it would be best to start with profiling and identify what the slowest parts of the program are.

In regards to OpenGL, as a Windows Forms control, the ImageBox won't work with it. You could in theory take a lot of code and dump into a OpenTK application (assuming they've finished cleaning house yet) and use that, but I don't know if it would be better to start from scratch rather than trying to hammer OpenGL into ImageBox logic (for example the different co-ordinate systems used by WinForms and OpenGL). I haven't used OpenTK for rendering huge graphics, but its a safe bet that it's going to perform much better than a WinForms program.

Not sure if this is viable, but WPF is hardware accelerated using DirectX, perhaps this is another option? I haven't used WPF in anger though so I don't have much to offer on how to do image processing using it.

A bit of a wishywashy post I'm afraid - but if you do some profiling and find that there's aspects of ImageBox causing you an issue if you can provide more information I could look into improving it.

Hope something in this helps!

Regards;
Richard Moss

[dr]

Hi,

Thanks for the quick response. I took your suggestion to profile the code (again) and found that I could eliminate the code listed above with just a few lines of Emgu/OpenCV code. I used one of their "Convert" functions to go from a one channel to a four channel Mat object(i.e. RGBG), then set the ImageBox.Image property equal to the Mat.Bitmap property. I can now get ~10 FPS, which will have to do in the short term. It does help to read the docs...

I will play with OpenTK or equivalent when I get a chance, to see what can be done in the future.

Thanks,

DR