A day with .Net

My day to day experince in .net

UI Automation Provider For a Custom Control

Posted by vivekcek on January 9, 2015

I hope the reader have sound knowledge about Microsoft UI Automation Framework (UIA).

Tools Required?

Visual Studio 2010/2012/2013.
Visual UI Spy (Link)
VIBlend Custom Grid Control (Link).

Libraries needed.

UIAutomationProvider.dll
UIAutomationTypes.dll
WindowsBase.dll

Look at the below screen, which contain two grid controls, one is a custom VIBlend grid ,another a default gridview in .NET.

Here is the Problem.

When we inspect both controls with Visual UI Spy,found that our custom VI Blend Grid control is identified as pane, also it’s rows and cells are not recognized.
But look at our default grid view, it is identified as table and its rows and cells are accessible.

1

Why this happen?

Because the default grid view come with .NET support UI automation, but custom controls may or may not support UI automation.

How to solve this?

Give server side UI automation support for the custom control (Refer MSDN if you don’t know about Server Side Provider).

Implementation

We use the concept of sub-classing here, as the source code of the custom control is not available with us.
We use this extended control in our forms.

Inside our custom control we override WndProc and catch the WM_GETOBJECT message and return our Server Side Provider implementation.

Code snippet to extend control.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Windows.Automation.Provider;
using System.Windows.Forms;

namespace VIBlendUIAutomation
{
    public class CustomGrid : VIBlend.WinForms.DataGridView.vDataGridView
    {
        private CustomGridProvider provider;

        protected override void WndProc(ref Message m)
        {
            const int WM_GETOBJECT = 0x003D;

            if ((m.Msg == WM_GETOBJECT) && (m.LParam.ToInt32() ==
                AutomationInteropProvider.RootObjectId))
            {
                m.Result = AutomationInteropProvider.ReturnRawElementProvider(
                        this.Handle, m.WParam, m.LParam,
                        this.Provider);
                return;
            }
            base.WndProc(ref m);
        }

        private CustomGridProvider Provider
        {
            get
            {
                if (this.provider == null)
                {
                    this.provider = new CustomGridProvider(this);
                }

                return this.provider;
            }
        }
    }
}

Code snippet to use the control in Form.

private CustomGrid vDataGridView1;
this.vDataGridView1 = new CustomGrid();

The core concept behind Server Side automation provider implementation is, convert the control into fragments and construct a tree.
Here we construct a tree with our Grid as root, rows and cells as it children.

Implementation of provider class, the important method here is Navigate(), which is responsible for constructing the tree.
This provider class implement ‘IRawElementProviderFragmentRoot’ , means provider is our root.
Provider also implement ‘IGridProvider’, which helps us to access cells with index.

Code snippet of provider class.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Windows.Automation;
using System.Windows.Automation.Provider;
using System.Windows.Forms;


namespace VIBlendUIAutomation
{
    public class CustomGridProvider : IRawElementProviderFragmentRoot, IGridProvider
    {
        private CustomGrid _grid;

        public CustomGridProvider(CustomGrid grid)
        {
            _grid = grid;
        }

        #region IRawElementProviderFragmentRoot

        public IRawElementProviderFragment ElementProviderFromPoint(double x, double y)
        {
            throw new NotImplementedException();
        }

        public IRawElementProviderFragment GetFocus()
        {
            throw new NotImplementedException();
        }

        public System.Windows.Rect BoundingRectangle
        {
            get
            {
                return System.Windows.Rect.Empty;
            }
        }

        public IRawElementProviderFragmentRoot FragmentRoot
        {
            get
            {
                return this;
            }
        }

        public IRawElementProviderSimple[] GetEmbeddedFragmentRoots()
        {
            List<IRawElementProviderSimple> embeddedFragmentRoots = new List<IRawElementProviderSimple>();
            for (int i = 0; i < this._grid.RowsHierarchy.Items.Count; i++)
            {
                embeddedFragmentRoots.Add(new CustomGridRow(this._grid, this, i));
            }

            return embeddedFragmentRoots.ToArray();

        }

        public int[] GetRuntimeId()
        {
            return null;
        }

        public IRawElementProviderFragment Navigate(NavigateDirection direction)
        {
            if (this._grid.RowsHierarchy.Items.Count > 0)
            {
                switch (direction)
                {
                    case NavigateDirection.FirstChild:
                        return new CustomGridRow(this._grid, this, 0);
                    case NavigateDirection.LastChild:
                        return new CustomGridRow(this._grid, this, this._grid.RowsHierarchy.Items.Count - 1);
                }
            }

            return null;
        }

        public void SetFocus()
        {
            throw new NotImplementedException();
        }

        public object GetPatternProvider(int patternId)
        {
            if (patternId.Equals(GridPatternIdentifiers.Pattern.Id) ||
                patternId.Equals(SelectionPatternIdentifiers.Pattern.Id))
            {
                return this;
            }
            return null;
        }

        public object GetPropertyValue(int propertyId)
        {
            if (propertyId == AutomationElementIdentifiers.ControlTypeProperty.Id)
            {
                return ControlType.DataGrid.Id;
            }
            else if (propertyId == AutomationElementIdentifiers.NameProperty.Id ||
                     propertyId == AutomationElementIdentifiers.AutomationIdProperty.Id)
            {
                return _grid.Name;
            }
            else if (propertyId == AutomationElementIdentifiers.IsKeyboardFocusableProperty.Id)
            {
                bool canFocus = false;
                _grid.Invoke(new MethodInvoker(() => { canFocus = _grid.CanFocus; }));
                return canFocus;
            }
            else if (propertyId == AutomationElementIdentifiers.ClassNameProperty.Id)
            {
                return this.GetType().ToString();
            }
            else if (propertyId == AutomationElementIdentifiers.IsEnabledProperty.Id)
            {
                return _grid.Enabled;
            }

            return null;
        }

        public IRawElementProviderSimple HostRawElementProvider
        {
            get
            {
                IntPtr hwnd = IntPtr.Zero;
                _grid.Invoke(new MethodInvoker(() => { hwnd = _grid.Handle; }));
                return AutomationInteropProvider.HostProviderFromHandle(hwnd);
            }
        }

        public ProviderOptions ProviderOptions
        {
            get
            {
                return ProviderOptions.ServerSideProvider | ProviderOptions.UseComThreading;
            }
        }

        #endregion

        #region IGridProvider
        public int ColumnCount
        {
            get { return this._grid.ColumnsHierarchy.Items.Count; }
        }

        public IRawElementProviderSimple GetItem(int row, int column)
        {
            if (this._grid.RowsHierarchy.Items.Count <= row)
            {
                throw new ArgumentException("Invalid row index specified.");
            }

            if (this._grid.ColumnsHierarchy.Items.Count <= column)
            {
                throw new ArgumentException("Invalid column index spceified.");
            }

            return new CustomGridCell(this._grid, this, row, column);
        }

        public int RowCount
        {
            get { return this._grid.RowsHierarchy.Items.Count; }
        }
        #endregion

        #region ISelectionProvider
        public bool CanSelectMultiple
        {
            get { return true; }
        }

        public IRawElementProviderSimple[] GetSelection()
        {
            List<IRawElementProviderSimple> selectedItems = new List<IRawElementProviderSimple>();

            foreach (var selectedItem in this._grid.RowsHierarchy.SelectedItems)
            {
                selectedItems.Add(new CustomGridRow(this._grid,this,selectedItem.ItemIndex));
            }

            return selectedItems.ToArray();
        }

        public bool IsSelectionRequired
        {
            get { return false; }
        }
        #endregion
    }
}

Rows are considered as the children of provider, This class implement two interfaces IRawElementProviderFragmentRoot and ISelectionItemProvider

Code snippet of custom row class.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Windows.Automation;
using System.Windows.Automation.Provider;

namespace VIBlendUIAutomation
{
    public class CustomGridRow : IRawElementProviderFragmentRoot, ISelectionItemProvider
    {
        private CustomGrid _grid;
        private int _rowIndex;
        protected IRawElementProviderFragment parent;
        protected IRawElementProviderFragmentRoot fragmentRoot;

        public CustomGridRow(CustomGrid grid, IRawElementProviderFragmentRoot root, int rowIndex)
        {
            _grid = grid;
            _rowIndex = rowIndex;
            this.parent = root;
            this.fragmentRoot = root;
        }

        #region IRawElementProviderFragmentRoot

        public IRawElementProviderFragment ElementProviderFromPoint(double x, double y)
        {
            throw new NotImplementedException();
        }

        public IRawElementProviderFragment GetFocus()
        {
            throw new NotImplementedException();
        }

        public System.Windows.Rect BoundingRectangle
        {
            get { return System.Windows.Rect.Empty; }
        }

        public IRawElementProviderFragmentRoot FragmentRoot
        {
            get { return this.fragmentRoot; }
        }

        public IRawElementProviderSimple[] GetEmbeddedFragmentRoots()
        {
            return null;
        }

        public int[] GetRuntimeId()
        {
            return new int[] { AutomationInteropProvider.AppendRuntimeId, _rowIndex };
        }

        public IRawElementProviderFragment Navigate(NavigateDirection direction)
        {
            switch (direction)
            {
                case NavigateDirection.FirstChild:
                    return new CustomGridCell(_grid, this, _rowIndex, 0);

                case NavigateDirection.LastChild:
                    return new CustomGridCell(_grid, this, _rowIndex, this._grid.ColumnsHierarchy.Items.Count - 1);

                case NavigateDirection.NextSibling:
                    if (_rowIndex < _grid.RowsHierarchy.Items.Count - 1)
                    {
                        return new CustomGridRow(_grid, this.fragmentRoot, _rowIndex + 1);
                    }
                    break;

                case NavigateDirection.PreviousSibling:
                    if (_rowIndex > 0)
                    {
                        return new CustomGridRow(_grid, this.fragmentRoot, _rowIndex - 1);
                    }
                    break;

                case NavigateDirection.Parent:
                    return this.parent;
            }

            return null;
        }

        public void SetFocus()
        {
            throw new NotImplementedException();
        }

        public object GetPatternProvider(int patternId)
        {

            if (patternId.Equals(SelectionItemPatternIdentifiers.Pattern.Id))
            {
                return this;
            }
            return null;
        }

        public object GetPropertyValue(int propertyId)
        {
            if (propertyId == AutomationElementIdentifiers.IsKeyboardFocusableProperty.Id)
            {
                return true;
            }
            else if (propertyId == AutomationElementIdentifiers.ClassNameProperty.Id)
            {
                return this.GetType().ToString();
            }
            else if (propertyId == AutomationElementIdentifiers.LocalizedControlTypeProperty.Id ||
                     propertyId == AutomationElementIdentifiers.ControlTypeProperty.Id)
            {
                return ControlType.DataItem.Id;
            }
            else if (propertyId == AutomationElementIdentifiers.NameProperty.Id)
            {
                return String.Format("Row {0}", _rowIndex.ToString());
            }
            else if (propertyId == AutomationElementIdentifiers.IsEnabledProperty.Id)
            {
                IRawElementProviderSimple provider = this.parent as IRawElementProviderSimple;
                return ((bool)provider.GetPropertyValue(AutomationElementIdentifiers.IsEnabledProperty.Id));
            }

            return null;
        }

        public IRawElementProviderSimple HostRawElementProvider
        {
            get { return null; }
        }

        public ProviderOptions ProviderOptions
        {
            get { return System.Windows.Automation.Provider.ProviderOptions.ServerSideProvider | System.Windows.Automation.Provider.ProviderOptions.UseComThreading; }
        }

        #endregion

        #region ISelectionItemProvider
        public void AddToSelection()
        {
            this._grid.RowsHierarchy.Items[_rowIndex].Selected = true;
        }

        public bool IsSelected
        {
            get { return this._grid.RowsHierarchy.Items[_rowIndex].Selected; }
        }

        public void RemoveFromSelection()
        {
            this._grid.RowsHierarchy.Items[_rowIndex].Selected = false;
        }

        public void Select()
        {
            this._grid.RowsHierarchy.Items[_rowIndex].Selected = true;
        }

        public IRawElementProviderSimple SelectionContainer
        {
            get { return this.parent; }
        }
        #endregion


    }
}

Cells are considered as the children of rows, This class implement the following interface IRawElementProviderFragment, IGridItemProvider, IValueProvider, ISelectionItemProvider.

Code snippet of custom cell class.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Windows.Automation;
using System.Windows.Automation.Provider;

namespace VIBlendUIAutomation
{
    public class CustomGridCell : IRawElementProviderFragment, IGridItemProvider, IValueProvider, ISelectionItemProvider
    {

        private CustomGrid _grid;
        private int _rowIndex;
        private int _cellIndex;
        protected IRawElementProviderFragment parent;
        protected IRawElementProviderFragmentRoot fragmentRoot;

        public CustomGridCell(CustomGrid grid, IRawElementProviderFragmentRoot root, int rowIndex, int cellIndex)
        {
            _grid = grid;
            _rowIndex = rowIndex;
            _cellIndex = cellIndex;
            this.parent = root;
            this.fragmentRoot = root;
        }

        #region IRawElementProviderFragment
        public System.Windows.Rect BoundingRectangle
        {
            get { return System.Windows.Rect.Empty; }
        }

        public IRawElementProviderFragmentRoot FragmentRoot
        {
            get { return this.fragmentRoot; }
        }

        public IRawElementProviderSimple[] GetEmbeddedFragmentRoots()
        {
            return null;
        }

        public int[] GetRuntimeId()
        {
            return new int[] { AutomationInteropProvider.AppendRuntimeId, _rowIndex, _cellIndex };
        }

        public IRawElementProviderFragment Navigate(NavigateDirection direction)
        {
            switch (direction)
            {
                case NavigateDirection.NextSibling:
                    if (_cellIndex < _grid.ColumnsHierarchy.Items.Count - 1)
                    {
                        return new CustomGridCell(_grid, this.fragmentRoot, _rowIndex, _cellIndex + 1);
                    }
                    break;
                case NavigateDirection.PreviousSibling:
                    if (_cellIndex > 0)
                    {
                        return new CustomGridCell(_grid, this.fragmentRoot, _rowIndex, _cellIndex - 1);
                    }
                    break;
                case NavigateDirection.Parent:
                    return new CustomGridRow(_grid, this.parent.FragmentRoot, _rowIndex);
            }

            return null;
        }

        public void SetFocus()
        {
            throw new NotImplementedException();
        }

        public object GetPatternProvider(int patternId)
        {

            if (patternId.Equals(GridItemPatternIdentifiers.Pattern.Id) ||
               patternId.Equals(ValuePatternIdentifiers.Pattern.Id) ||
               patternId.Equals(SelectionItemPatternIdentifiers.Pattern.Id))
            {
                return this;
            }
            else
            {
                return null;
            }

        }

        public object GetPropertyValue(int propertyId)
        {
            if (propertyId == AutomationElementIdentifiers.ControlTypeProperty.Id ||
               propertyId == AutomationElementIdentifiers.LocalizedControlTypeProperty.Id)
            {
                return ControlType.DataItem.Id;
            }
            else if (propertyId == AutomationElementIdentifiers.AutomationIdProperty.Id)
            {
                return "Cell[" + _rowIndex.ToString() + "][" + _cellIndex.ToString() + "]"; ;
            }
            else if (propertyId == AutomationElementIdentifiers.NameProperty.Id)
            {
                return this._grid.CellsArea.GetCellValue(this._grid.RowsHierarchy.Items[_rowIndex], this._grid.ColumnsHierarchy.Items[_cellIndex]);
            }
            else if (propertyId == AutomationElementIdentifiers.ClassNameProperty.Id)
            {
                return this.GetType().ToString();
            }
            else if (propertyId == AutomationElementIdentifiers.HasKeyboardFocusProperty.Id)
            {
                return false;
            }
            else if (propertyId == AutomationElementIdentifiers.IsEnabledProperty.Id)
            {
                return _grid.Enabled;
            }
            else if (propertyId == AutomationElementIdentifiers.IsKeyboardFocusableProperty.Id)
            {
                return _grid.Enabled && _grid.Visible;
            }

            return null;
        }

        public IRawElementProviderSimple HostRawElementProvider
        {
            get { return null; }
        }

        public ProviderOptions ProviderOptions
        {
            get { return System.Windows.Automation.Provider.ProviderOptions.ServerSideProvider | System.Windows.Automation.Provider.ProviderOptions.UseComThreading; }
        }
        #endregion

        #region IGridItemProvider
        public int Column
        {
            get { throw new NotImplementedException(); }
        }

        public int ColumnSpan
        {
            get { throw new NotImplementedException(); }
        }

        public IRawElementProviderSimple ContainingGrid
        {
            get { throw new NotImplementedException(); }
        }

        public int Row
        {
            get { throw new NotImplementedException(); }
        }

        public int RowSpan
        {
            get { throw new NotImplementedException(); }
        }
        #endregion

        #region IValueProvider
        public bool IsReadOnly
        {
            get { return false; }
        }

        public void SetValue(string value)
        {
            this._grid.CellsArea.SetCellValue(this._grid.RowsHierarchy.Items[_rowIndex], this._grid.ColumnsHierarchy.Items[_cellIndex], value);
        }

        public string Value
        {
            get { return this._grid.CellsArea.GetCellValue(this._grid.RowsHierarchy.Items[_rowIndex], this._grid.ColumnsHierarchy.Items[_cellIndex]).ToString(); }
        }
        #endregion

        #region ISelectionItemProvider
        public void AddToSelection()
        {
            this._grid.ColumnsHierarchy.Items[_cellIndex].Selected = true;
        }

        public bool IsSelected
        {
            get { return this._grid.ColumnsHierarchy.Items[_cellIndex].Selected; }
        }

        public void RemoveFromSelection()
        {
            this._grid.ColumnsHierarchy.Items[_cellIndex].Selected = false;
        }

        public void Select()
        {
            this._grid.ColumnsHierarchy.Items[_cellIndex].Selected = true;
        }

        public IRawElementProviderSimple SelectionContainer
        {
            get { return this.parent.FragmentRoot; }
        }
        #endregion
    }
}

Now look at the pic below, UI Spy recognize our custom grid as grid and its rows and cells are listed.

Capture

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s