As I created myself VFIO-MDEV driver/virtual device, the next steps I can think are:

1) make virtual machine can access real hardware through this VFIO-MDEV driver.

2) hardware/VFIO-MDEV driver can access the buffer in virtual machine. Because many HW will write/read data from buffer/memory. If we vituralize a HW and provide its virtual device to the virtual machine, we will need this feature.

So this blog will introduce:

1) How does virtual machine access HW(VFIO-MDEV virtual device's) registers.

2) How does VFIO-MDEV driver access virtual machine's buffer/memory.

1. How does virtual machine access HW registers.

On host VFIO-MDEV driver:

Firstly I fixed a bug on virtual device's PCI configure space. That is the PCI device class. Last time I set it to 0x0. This will cause pci_enable_device() error. So I modified it to 0xFF(misc device).

Then I need to prepare the code of receiving the registers in handle_bar_write(). The virtual machine will write registers to BAR0 IO mapping space(because BAR0 is IO mapped in VFIO-MDEV driver). And my VFIO-MDEV driver(mdev_parent_ops.write()) will accept the register value and pass it to handle_bar_write().

My code accept two 4 bytes address and put them together to become a 8 bytes address.

On virtual machine:

In my driver(yes, this is the driver on virtual machine for VFIO-MDEV virtual device) I added a function to write the BAR register.

And I called it in probe function.

Result:

2. How does VFIO-MDEV driver access virtual machine's buffer/memory.

KVM actually has many APIs to access guest, such as kvm_read_guest() and kvm_write_guest(). However they all need a parameter of structure kvm. So I need to get virtual machine's kvm structure. How? Luckly I found VFIO provides a good way for us to get the kvm structure.

On host VFIO-MEDV driver:

My driver need to register a notifier to VFIO. And when the virtual machine attaches the virtual device, VFIO will nofity the virtual machine's kvm structure to my driver.

After I get the kvm structure, I can use it to read the virtual machine's buffer. You can see the virtual machine's buffer address (GPA--guest physical address)is from register writing.

On virtual machince:

My driver malloc a buffer and fill a sting "vm_buffer_test" into the buffer. Then write the buffer's physical address to virtual device.

Result:

Tips:

1. PCI IO mapping issue.

The original mtty driver designed two BARs(each BAR has 8 registers, 8 bytes space). And when I tried to enlarge the BAR space, my host VFIO-MDEV driver(virtual device attach to virtual machine) will crash the virtual machine. Actually I don't know the root cause but the virtual machine was killed.

Then I just enable one BAR and enlarge it's space to the whole PCI bridge space(0x1000). It works good.

So it's a pending issue.

2. outl() just output 4 bytes.

And if your IO space is not enough for 4 bytes, it may cause wrong action on KVM. I found this by made a mistake that write 4 bytes to reg address 0x6 when the only 0x8 register space. I only see 2 bytes arrived to host driver.